A methodological and reporting quality assessment of systematic reviews/meta-analyses on exercise interventions for cognitive function in older adults with mild cognitive impairment

Search strategy

Two independent investigators (WZ, NX) performed comprehensive literature searches in five pertinent databases: PubMed, Cochrane Library, Embase, Scopus, Physiotherapy Evidence Database and Web of Science. The search time limit was set from the inception of each database to June 1, 2024. The search terms used included: (“Exercise”[Mesh] OR “exercise” OR “physical activity” OR “resistance training” OR “strength” OR “endurance” OR “walking” OR “yoga” OR “TaiChi”) AND (“Cognitive Dysfunction”[Mesh] OR “mild cognitive impairment” OR “MCI”) AND (“systematic review” OR “meta-analysis”) NOT (“Alzheimer’s disease” OR “AD”).

Inclusion and exclusion criteria

Literature selection and data extraction

Two investigators (WZ, NX) independently screened and organized the literature and extracted relevant information. The selection process initially involved reviewing the titles and abstracts, followed by cross-checking the extracted data. In cases of disagreement, a third investigator (MF) assisted in the decision-making process. After excluding irrelevant literature through preliminary screening, the full text of the remaining articles was reviewed for final inclusion. The following data were extracted: (1) basic information, such as study title, authors, publication date, journal, number of original studies included, sample size, intervention details, outcome measures, main conclusions, quality assessment tools, meta-analysis registration numbers, and funding sources; (2) methodology for literature evaluation and reporting quality-related content; and (3) qualitative or quantitative results for each outcome measure. Primary outcome measures included the Mini-Mental State Examination (MMSE), Alzheimer’s Disease Rating Scale-Cognitive Subscale (ADAS-cog), and Montreal Cognitive Assessment (MoCA).

Methodological quality evaluation

Two researchers (WZ, NX) independently assessed the methodological quality of the included studies using the AMSTAR2 scale, which comprises 16 items. Each item was rated as “yes,” “no,” or “partially yes” based on the extent to which evaluation criteria were satisfied. Items 2, 4, 7, 9, 11, 13, and 15 are considered critical. Studies were assigned one of four credibility ratings: “High,” “Moderate,” “Low,” or “Very Low.” A rating of “High” was given if a study had 0 or 1 non-critical flaws, “Moderate” if it had more than one non-critical flaw, “Low” if it had one critical flaw with or without non-critical flaws, and “Very Low” if it had more than one critical flaw. Disagreements during the evaluation process were resolved through discussions with a third(MF) researcher.

Literature screening process and results

Eighteen meta-analyses were ultimately included in the analysis after screening 1,762 articles based on specified criteria. Figure 1 illustrates the literature screening process. The list of references deleted after reading the full text can be found in the Appendix S1.

Basic characteristics of the included literature

The included meta-analyses were published between 2018 and 2023, and all the original studies were randomized and controlled. The largest study included 2,526 participants and the smallest had 182 participants, involving a total of 18,461 individuals. The interventions in the trial groups consisted of aerobic, physical, mental, resistance, and multipart exercises. The control group in the randomized trials maintained their regular daily activities without any specific exercise interventions. Regarding methodological quality assessment tools, 12 studies utilized the Cochrane Collaboration Network RoB risk assessment tool, four employed the PEDro scale, one used the Cochrane Collaboration Network RoB and PEDro, and one applied the EPHPP for risk assessment. All 18 studies consistently indicated that exercise effectively improves cognition in older adults with MCI. The basic characteristics of the included studies are presented in Table 1.

Methodological quality evaluation of the included literatures

The methodological quality of the 18 included meta-analyses was assessed using the AMSTAR2 scale. The results showed that, in terms of quality, one study was rated as ‘moderate’, seven were rated as ‘low’, and ten were rated as ‘very low’. The detailed results are presented in Table 2.

Evaluating the impact of exercise interventions on cognitive assessment scores in mild cognitive impairment

Methodological quality assessment of the meta-analytic literature

Methodological quality assessment of the 18 meta-analyses using the AMSTAR2 scale revealed that the overall methodological quality of the included trials was low. This indicated that individual meta-analytic studies had varying degrees of methodological deficiencies in terms of topic selection, design, registration, data extraction, and statistical analysis. These deficiencies are mainly attributed to the following aspects.

(1) The included studies did not explain whether they had drafted a research proposal, and only a few studies were registered on the systematic review platform in advance. To effectively reduce the risk of selective reporting bias and avoid being affected by arbitrary decisions in data extraction and use, it is necessary to write a plan or pre-register in advance. In addition, registration on registration platforms (such as The Cochrane Library and PROSPERO) is also conducive to improving the transparency of meta-analysis research.

(2) None of the included studies provided a detailed description of the reasons for excluding the literature. Although the included studies reported a two-person search, screening, and data extraction, it is unknown whether this was performed with strict compliance, which increases the risk of bias. It is recommended that journals require authors to provide a list of excluded studies to ensure transparency during the screening process.

(3) The meta-analysis implementation process was not sufficiently rigorous. Seventeen studies did not adequately consider the potential impact of the risk of bias in the included studies on the analysis of results and did not provide satisfactory answers to the heterogeneity in the meta-analysis results. Additionally, three studies did not adequately evaluate publication bias. Meta-analysis is recognized as the best method by which research evidence on a particular issue can be objectively evaluated and synthesized. The risk of bias analysis, interpretation of sources of heterogeneity, and evaluation of publication bias during the implementation of the meta-analysis determined the reliability of the results. When randomized controlled trials or non-randomized intervention studies with different risks of bias are included, the investigator should conduct an in-depth analysis of the risk of bias in the included literature and discuss its impact on the results (qualitative analysis results or pooled effect sizes). In addition, when heterogeneity tests suggest the presence of heterogeneity, researchers should make every effort to find potential causes of heterogeneity and explore the probable causes of heterogeneity in the discussion section.

(4) Ten studies did not specify the funding sources for the studies included in the meta-analysis production process, which may have led to potential bias, opacity, and weakened trust. The source of funding for a study may have an impact on the findings, as funders may have their own interests and preferences. Lack of information on the source of funding may cause readers to doubt the reliability and transparency of the study. Knowing the source of funding can help readers to understand the context of the study and the interests of the researcher. Simultaneously, if readers suspect that the results of the study are influenced by the source of funding, they may be skeptical about the results of the study and reduce trust in the study.

These findings have significant implications for future research methodologies and clinical practices in exercise interventions for MCI. Our results underscore the importance of adhering to rigorous research protocols and pre-registering studies to enhance transparency and replicability in the design and implementation of exercise interventions for MCI. Detailed documentation of excluded literature and clear disclosure of funding sources will contribute to the credibility of the research and public trust in the findings. More stringent analyses of bias risk and heterogeneity in the studies included in the meta-analysis will improve the accuracy and practicality of exercise intervention recommendations in clinical practice.

Interpretation of heterogeneity in included studies

In conducting the methodological quality assessment of the included meta-analyses, we observed significant heterogeneity among the studies, which notably impacted our evaluation results. The heterogeneity manifested in several key aspects.

(1) Diversity in study design

The meta-analyses varied in their underlying study designs, encompassing differences in the types of exercise interventions, baseline cognitive status of participants, and the duration and frequency of the interventions. This diversity could lead to varying interpretations of the effects of exercise interventions.

(2) Participant characteristics

Variations in participant demographics such as age, gender, and stage of disease could influence the effectiveness of the exercise interventions. Different studies may focus on distinct populations, affecting the perceived efficacy of exercise on cognitive functions.

(3) Methodological variability

Inconsistencies in data extraction, statistical analysis approaches, and bias risk assessment across studies could contribute to variability in conclusions. For instance, the methods for handling missing data and the choice of effect size models varied, potentially leading to different outcomes.

In summary, the heterogeneity of the studies played a significant role in our methodological assessment. Future methodological evaluations of similar meta-analyses should delve deeper into these heterogeneity factors, detailing them in the study design and reporting to enhance the clarity and comprehensiveness of the findings.

Recommendations for improvement

Based on the results of this study, the author makes the following recommendations were made to inform future research and evaluation criteria:

(1) Meta-analysis authors should adhere to the appropriate guidelines for study design, implementation, and reporting. Essential strategies include (a) Consulting the Cochrane Handbook for Systematic Reviews of Interventions to grasp meta-analysis fundamentals, applications, and advantages/disadvantages of relevant software (e.g., RevMan, Stata, R, CMA, and OpenMetaAnalyst). For instance, despite RevMan’s suitability for exercise intervention trials, it has limitations in publication bias testing, and authors should be familiar with comprehensive tests such as Egger’s regression analysis and Begg’s rank correlation method available in CMA and Stata. Additionally, increasing the pre-registration of protocols is vital. Authors are encouraged to pre-register their study protocols in registries such as PROSPERO, detailing the planned methods and analyses to enhance transparency and aid in reducing publication bias. Furthermore, enhancing the thorough reporting of study screening and reasons for exclusion is critical for improving transparency and reproducibility. Authors must provide detailed accounts of study selection processes, including clear documentation of screening procedures, criteria for inclusion and exclusion, and explicit reasons for excluding studies, especially those that might influence the meta-analysis outcomes. Better handling of heterogeneity is also essential. Authors should not only report the presence of heterogeneity but explore and explain its sources. This can be achieved by conducting subgroup analyses or meta-regression to investigate potential moderators of effect sizes. A detailed interpretation of how variations between studies might impact the overall findings and conclusions will significantly enhance the quality and credibility of the meta-analysis. (b) Refining and pre-registering study plans based on AMSTAR2 and PRISMA checklists, ensuring detailed reporting, particularly in the AMSTAR2 Methods section, to minimize bias. (c) Emphasizing both the theoretical and practical implications of meta-analyses, high-quality studies should provide reliable evidence, adhere to reporting standards, and advance research fields while offering practical guidance.

(2) To improve the quality of meta-analyses, journal editorial boards and departments should strengthen the reporting requirements through review policies. We suggest the following measures: (a) Develop self-checking report guidelines for meta-analysis authors based on AMSTAR2 and PRISMA checklists tailored to specific situations and establish meta-analysis topics to enhance their impact and standardization. (b) Journal editors should refer to meta-analysis reporting guidelines when initially screening manuscripts or conducting peer-expert reviews to ensure the accuracy and coherence of the report. (c) It should be noted that the AMSTAR2 and PRISMA checklists mainly apply to trial (intervention)-type meta-analyses; some entries may not be applicable to meta-analyses of sports science correlation studies.

(3) Universities need to enhance their theoretical and practical courses on systematic reviews and meta-analyses. While some medical schools or comprehensive colleges have already introduced “evidence-based medicine” or “systematic review/meta-analysis” courses, sports colleges often lack such offerings, hindering the development of professionals in the field. Therefore, we suggest that sports colleges and universities incorporate meta-analysis theory and practice courses into their postgraduate training programs or integrate relevant lectures on systematic review/meta-analysis topics into existing “research methods” and “statistics” courses. This approach can better cultivate professionals with systematic review/meta-analysis expertise and improve the overall educational quality.

Limitations

There are some limitations to this study: (1) only English literature was included, and the search results may not be comprehensive; (2) there is inevitably some subjectivity in the evaluation and analysis process, which may also lead to bias; (3) a lack of discrimination capacity of the AMSTAR 2.0 tool may affect the ability to achieve moderate or high methodological quality in systematic reviews; (4) the NCSE and COR-SOG scales were only administered once, and the results should be treated with caution.