Several methods for assessing research waste in reviews with a systematic search: a scoping review

Protocol and eligibility criteria

We used a scoping review approach to investigate the methods used in reviews based on a systematic search for papers that examined or discussed research waste, following the guidance for scoping reviews previously established (Munn et al., 2018). We reported our study following the Preferred Reporting Items for Systematic reviews and Meta-Analysis extension for Scoping Reviews (PRISMA-ScR) reporting guideline (Tricco et al., 2018) and defined the eligibility criteria according to the Joanna Briggs Institute guidance (Peters et al., 2021). The protocol was registered at Open Science Framework (Rosengaard et al., 2023). This scoping review reports on a prespecified secondary outcome in the protocol, whereas the main outcome is reported elsewhere (Rosengaard et al., 2024b).

Utilizing the population, concept, and context framework for scoping reviews (Peters et al., 2021), the concept examined was how research waste and related synonyms (e.g., unnecessary, redundant, duplicate, etc.) were assessed. We included reviews that employed a systematic search strategy, as a search process that is comprehensive, structured, and replicable, intended to identify all relevant literature on a given topic. This, e.g., includes systematic reviews, scoping reviews, and overviews of reviews, all of which must have a clearly documented search strategy to be considered for inclusion. Reviews were included if they defined or discussed research waste or its synonyms as part of their focus. The context of our scoping review related to any method reported to be used to examine research waste in their included reviews for different study designs. We included reviews published in scholarly journals with systematic searches in all languages from the last 30 years (1993–2023) to provide a contemporary view of the subject. We chose a 30-year timeframe to review recent advances in research waste assessment methods. We excluded reports that did not include a definition or discussion of research waste or related terms or synonyms. Furthermore, we excluded reviews that did not directly examine research waste, reviews of veterinary sciences, reviews that did not analyze health-related research on humans, and titles marked as retracted.

Information sources and search

Through a discussion of examples of research waste in the author group, several ways to describe the concept were agreed upon. Pilot searches were conducted to find synonyms of research waste based on these wordings. The final search string was developed in PubMed and adapted for Embase. It was consulted with and approved by an information specialist. The last date of the search was August 17, 2023. The full search strategy is presented in the protocol (Rosengaard et al., 2023). We performed a backward citation search (Greenhalgh & Peacock, 2005), and the final included reviews were crosschecked in the Retraction Watch database (Crossref, 2024).

Selection and data

Two reviewers independently screened records in Covidence (Veritas Health Innovation, 2024), initially on title and abstract and subsequently in full text according to the eligibility criteria. Conflicts were resolved through consensus discussions. Data extraction was performed independently in duplicate. Data were extracted to pilot forms in Microsoft Excel (Microsoft Corporation, Redmond, WA, USA). We extracted the following data from the included reviews: type of review with a systematic search (systematic review, scoping review, or overview of reviews), number of included studies in the reviews, the study designs that the reviews included, the aim of the reviews, which aspects of research waste that were examined according to the MINUS aspect framework (Rosengaard et al., 2024b), and the methods reported by authors as used for investigating research waste. During data extraction, we retrospectively assigned each research waste assessment method to a waste category defined by the MINUS framework (Rosengaard et al., 2024b).

The aspects of research waste as defined by MINUS are Methodological, Invisible, Negligible, Underreported, and Structural research waste (Rosengaard et al., 2024b). Methodological research waste includes waste related to study conduct, methodological quality, and recruitment and retention of patients. Invisible research waste concerns a lack of data sharing, non-publication of conducted work, inaccessible research, and discontinuation of trials. Negligible research waste includes unnecessary duplication of research work and unjustified research, e.g., lack of prior literature search. Underreported research waste especially concerns reporting guideline adherence, written reporting, and heterogeneous outcome reporting. Finally, structural research waste concerns prioritization in research, implementation in clinical settings, inequity in health, patient involvement, management, and collaboration. We synthesized the results in a descriptive analysis, and the tools used in the included reviews are presented in tables and figures. We utilized an UpSet plot (Conway, Lex & Gehlenborg, 2017) to visualize overlapping aspects when examining research waste as the five aspects of MINUS (Rosengaard et al., 2024b). We categorized the different tools used to evaluate research waste according to which study designs the reviews included in their examination (Pollock et al., 2023). All extracted data and excluded studies are shared (Rosengaard et al., 2024a).

Selection and characteristics

The study selection process is presented in Fig. 1. The search identified 4,285 records of which 93 reviews were included in this scoping review (Pussegoda et al., 2017a; Clyne et al., 2020; Ker & Roberts, 2015; Ndounga Diakou et al., 2017; Sheth et al., 2011; Wu et al., 2022; Arundel & Mott, 2023; Kostalova et al., 2022; Limones et al., 2022; Doumouchtsis et al., 2019; Habre et al., 2014; Webbe et al., 2020; Pergialiotis et al., 2018; Créquit et al., 2016; Hacke & Nunan, 2019; Martel et al., 2012; Pussegoda et al., 2017b; Reddy et al., 2023; Slattery, Saeri & Bragge, 2020; Townsend et al., 2019; Synnot et al., 2018; Choi et al., 2022; Whear et al., 2022; Bendersky et al., 2023; Frost et al., 2018; Chambers et al., 2014; Hancock & Mattick, 2020; Houghton et al., 2020; Collins & Lang, 2018; Patarčić et al., 2015; Pandis et al., 2021; Fisher et al., 2022; Bolland, Avenell & Grey, 2018; Ramke et al., 2018; Sebastianski et al., 2019; Amad et al., 2019; Clarke, Brice & Chalmers, 2014; Avau et al., 2023; Xu et al., 2021; Mikelis & Koletsi, 2022; Ahmed Ali et al., 2018; Papathanasiou et al., 2016; Mercieca-Bebber et al., 2016; Gale et al., 2013; Shepard et al., 2023; Wright et al., 2000; Briel et al., 2016; Grégory et al., 2020; Sauzet, Kleine & Williams, 2016; Morgan et al., 2021; Siemens et al., 2022; Hey et al., 2017; Palmer et al., 2018; Blanco-Silvente et al., 2019; Cook, 2014; Maxwell et al., 2023; Sawin & Robinson, 2016; Lund et al., 2022; Sharma et al., 2019; Bolland, Grey & Avenell, 2018; Johnson et al., 2020; Torgerson et al., 2020; Gysling, Khan & Caruana, 2023; McGill et al., 2020; Agbadjé et al., 2022; Meneses-Echavez et al., 2019; Page et al., 2016; Mercieca-Bebber et al., 2022; Andaur Navarro et al., 2022; Dhiman et al., 2021; Okomo et al., 2019; Feng et al., 2022; Yu et al., 2018; Harman et al., 2021; Pascoe et al., 2021; Duffy et al., 2017; Evans et al., 2020; Velde et al., 2021; Rives-Lange et al., 2022; Cirkovic et al., 2020; Butcher et al., 2020; Dal Santo et al., 2023; Bero, Chiu & Grundy, 2019; Boutron & Ravaud, 2018; Klaic et al., 2022; Holmes et al., 2020; Cruz Rivera et al., 2017; Levati et al., 2016; Albarqouni, Elessi & Abu-Rmeileh, 2018; Bentley et al., 2019; Coffey et al., 2022; Tybor et al., 2018; Nankervis, Maplethorpe & Williams, 2011). Characteristics of the included, reviews are presented in Table 1. Of the included reviews, the majority were systematic reviews (n = 73). The reviews examined a median of 90 (10–6,781) studies for research waste. The included reviews’ most frequently examined study designs were randomized controlled trials (RCTs) (47%) and systematic reviews or meta-analyses (33%) (Table 1). About one-fourth (23%) of the reports included multiple types of study design. In the last ten years, the number of reports assessing research waste has increased (Table 1).

Aspects of research waste being examined

Figure 2 shows which MINUS aspects of research waste that was examined by the included reviews, and that about half of the reviews examine multiple aspects of research waste. Methodological research waste was the most assessed aspect (n = 51). Here, reviews mainly focused on the assessment of conduct examination and methodological quality assessment (Fig. 3). Conduct examination refers to assessing how the research was conducted and identifying potential practices that may have led to waste. Quality assessment refers to evaluating the methodological quality or rigor of the included reviews, focusing on how well the studies were conducted and reported. Underreported (n = 42) and structural (n = 22) research waste were also commonly investigated aspects (Fig. 2).

Across the included study designs

Figure 3 shows each aspect of MINUS and what the reviews examined within these aspects. When examining each research waste aspect of MINUS, regardless of study design, they included the following evaluations.

Methodological research waste was evaluated through quality assessment (32%) or conduct examination (21%) (Fig. 3). When assessing methodological quality, specific tools were used, primarily the risk of bias tools that are specific to the design of the review’s included studies (Table 2). Two reports used self-constructed risk of bias tools to assess for quality assessments (Collins & Lang, 2018; Patarčić et al., 2015).

Invisible research waste was mainly evaluated concerning inadequate data sharing (6%), non-publication (5%), and discontinuation (3%) (Fig. 3). Only one review (Mercieca-Bebber et al., 2016) used a tool to examine invisible research, i.e., the Framework Method for analyzing qualitative data (Gale et al., 2013) to account for missing data. Non-publication was primarily examined by checking the publication status of reports in registries such as ClinicalTrials.gov (Shepard et al., 2023).

Negligible research waste was assessed by examining unnecessary duplication of research effort (12%) or by reviewing the included reports for whether they had adequately justified conducting their research (10%) (Fig. 3 and Table 3), e.g., by checking the introduction for assessment of prior literature within the topic (Ker & Roberts, 2015; Sheth et al., 2011; Habre et al., 2014; Martel et al., 2012; Chambers et al., 2014; Mikelis & Koletsi, 2022; Bolland, Grey & Avenell, 2018; Johnson et al., 2020; Torgerson et al., 2020).

Underreported research waste was examined either through reporting guideline adherence (26%), heterogeneous outcome reporting (12%), or the written reporting (8%) (Fig. 3). The most common reporting guidelines utilized are displayed in Table 4. Reporting of outcomes was primarily examined when creating a core outcome set.

Structural research waste was mainly evaluated by examining prioritization (11%) or implementation (7%) (Fig. 3). When examining prioritization in the research field, evidence maps were commonly used to create an overview of the subject and where future research should be prioritized (Synnot et al., 2018; Choi et al., 2022; Bendersky et al., 2023; Avau et al., 2023; Rives-Lange et al., 2022; Coffey et al., 2022; Tybor et al., 2018; Nankervis, Maplethorpe & Williams, 2011). Implementation could be measured by implementation frameworks, where one report found 24 different implementation frameworks used to measure research impact (Cruz Rivera et al., 2017). The included reviews used different frameworks, but for the mixed study designs the Conceptual Framework for Implementation Fidelity (CFIF) (Agbadjé et al., 2022; Holmes et al., 2020) and the related Conceptual Framework for Implementation Research (CFIR) (Holmes et al., 2020) were used (Table 5). Thirty-three reports used a descriptive analysis to examine for research waste. These are categorized in Table 6 (Clyne et al., 2020; Limones et al., 2022; Habre et al., 2014; Slattery, Saeri & Bragge, 2020; Pandis et al., 2021; Fisher et al., 2022; Bolland, Avenell & Grey, 2018; Ramke et al., 2018; Sebastianski et al., 2019; Clarke, Brice & Chalmers, 2014; Papathanasiou et al., 2016; Briel et al., 2016; Grégory et al., 2020; Sauzet, Kleine & Williams, 2016; Morgan et al., 2021; Siemens et al., 2022; Hey et al., 2017; Palmer et al., 2018; Lund et al., 2022; Sharma et al., 2019; Pascoe et al., 2021; Duffy et al., 2017; Evans et al., 2020; Velde et al., 2021; Rives-Lange et al., 2022; Cirkovic et al., 2020; Butcher et al., 2020; Dal Santo et al., 2023; Bero, Chiu & Grundy, 2019; Boutron & Ravaud, 2018; Cruz Rivera et al., 2017; Levati et al., 2016; Albarqouni, Elessi & Abu-Rmeileh, 2018; Bentley et al., 2019).

Study design and tools to assess research waste

Systematic reviews and meta-analysis

A total of 31 reviews focused on systematic reviews with or without meta-analyses. Methodological research waste was evaluated by examination of conduct and quality assessment. Examination of conduct was done by analyzing power (Amad et al., 2019; Clarke, Brice & Chalmers, 2014), search and screening (Avau et al., 2023), statistical analyses (Fisher et al., 2022; Xu et al., 2021), and whether the reviews had published a protocol (Mikelis & Koletsi, 2022). Tools for quality assessment are displayed in Table 2 along with their count. The most used tools for quality assessment were A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 1 (Créquit et al., 2016; Hacke & Nunan, 2019; Martel et al., 2012; Pussegoda et al., 2017b; Reddy et al., 2023; Slattery, Saeri & Bragge, 2020; Townsend et al., 2019; Synnot et al., 2018), AMSTAR 2 (Choi et al., 2022; Whear et al., 2022; Bendersky et al., 2023), and Overview Quality Assessment Questionnaire (OQAQ) (Sheth et al., 2011; Pussegoda et al., 2017b; Reddy et al., 2023) (Table 4). Assessment of negligible research waste in systematic reviews was either focused on prior literature assessment or by exploring overlap among different reviews. Prior literature assessment was evaluated by Robinson’s Prior Research Citation Index (RPRCI) (Sawin & Robinson, 2016) or Robinson’s Sample Size Citation Index (RSSCI) (Sawin & Robinson, 2016). Overlap could be calculated using the Corrected Covered Area (CCA) (Reddy et al., 2023), by network analysis (Whear et al., 2022), or with a Sankey diagram (Maxwell et al., 2023) (Table 3). Reporting guideline adherence among systematic reviews and meta-analyses was primarily checked using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline (Pussegoda et al., 2017a; Hacke & Nunan, 2019; Pussegoda et al., 2017b; Whear et al., 2022; Fisher et al., 2022; Page et al., 2016) or Quality of Reporting of Meta-analyses (QUOROM) (Hacke & Nunan, 2019; Pussegoda et al., 2017b; Reddy et al., 2023). Other checklists used are listed in Table 4. One report examined guideline documentation (Mikelis & Koletsi, 2022) and another examined protocol adherence (Page et al., 2016). The Framework of Implementability (Klaic et al., 2022) was used to evaluate the impact of the review when published (Table 5).

Observational studies

A total of 15 reviews focused on observational studies. Methodological quality among observational studies was evaluated using the Newcastle-Ottawa scale risk of bias tool (Kostalova et al., 2022) (Table 2) or by the conduct of statistical tests (Cirkovic et al., 2020) or sample size calculations (Feng et al., 2022). Underreported research waste was assessed by reporting guideline adherence to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) statement (Andaur Navarro et al., 2022; Dhiman et al., 2021), The STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) guideline (Page et al., 2016; Okomo et al., 2019), or Standards for Reporting Diagnostic Accuracy (STARD) (Feng et al., 2022) (Table 4).

Non-randomized controlled trials

A total of 14 reviews focused on non-randomized trials. When checking trials for reporting guideline adherence, two reports used the TIDieR checklist (Agbadjé et al., 2022; Yu et al., 2018) (Table 4). Reporting of outcomes was categorized by the Core Outcome Measures in Effectiveness Trials (COMET) taxonomy (Harman et al., 2021). One report used the Accumulating Evidence and Research Organization (AERO) model (Patarčić et al., 2015) to prioritize the research subject when planning translational research.