Comparative analysis of deep learning algorithms for dental caries detection and prediction from radiographic images: a comprehensive umbrella review

Study design

This umbrella review uses SRs/MAs and carefully selected original studies (with specified inclusion criteria) to examine the accuracy, sensitivity, and specificity of various DL and CNN algorithms in detecting DC from radiographic images. The study methodology adheres to the “Reporting guideline for overviews of reviews of healthcare interventions” (PRIOR). Additionally, the preferred reporting items for systematic reviews and meta-analyses (PRISMA) flow diagram was used to illustrate the flow of study selection (Page et al., 2021; Gates et al., 2022).

Study type

This overview focuses specifically on SRs/MAs of original studies using various DL and CNN algorithms to predict DC from radiographic images. Table 1 provides a detailed summary of the research question addressed.

Inclusion criteria

• 1)

  SRs comprising a minimum of two primary studies.

• 2)

  Focusing on the assessment of DL or CNN algorithm accuracy in detecting DC using radiographic images.

• 3)

  SR articles published up to August 2023.

Exclusion criteria

• 1)

  Conference proceedings, narrative review articles, and scientific posters available online.

• 2)

  SRs or MAs with ambiguous inclusion/exclusion criteria.

• 3)

  Studies presenting results outside the scope of this research.

• 4)

  SRs without a data extraction table or measurable outcome reporting.

Search strategy

The search process involved querying electronic databases such as Medline via PubMed, Scopus, Web of Science, and Embase for articles published up to the end of August 2023. Additionally, we reviewed articles published in languages other than English, translating them if necessary for inclusion. A comprehensive search syntax was developed using carefully selected keywords in different combinations with Boolean operators, and relevant MeSH terms were applied for PubMed. This syntax was adapted for other databases, following each database’s specific guidelines to ensure search accuracy and compatibility. Details regarding the number of results retrieved from each database are provided in Table 2.

Study selection and data extraction

Endnote X20 (Clarivate, Philadelphia, PA, USA) was used for citation management. After removing duplicate articles, two reviewers (H.A. and F.Kh.) performed the initial title and abstract screening phase. Any discrepancies were resolved through consultation with a third reviewer (J.L.). Subsequently, both reviewers independently evaluated the full texts of eligible studies, adhering to predetermined inclusion and exclusion criteria. Any disagreements during this phase were resolved through further discussions with the third reviewer. The PRISMA flow diagram depicting this process is displayed in Fig. 1.

Data extraction process

Two researchers (M.A. and Z.Kh.) were responsible for extracting data from the selected studies, while another pair (N.Z. and M.S.) verified all extracted data. Any disagreements were resolved through consensus, facilitated by a third reviewer (J.L.).

Table 3 summarizes the SRs and MAs included in this overview. The data extraction checklist includes the author’s name, year of publication, number of relevant articles included, databases searched, inclusion and exclusion criteria, and main outcomes.

Table 4 presents the data extracted from each included original article, encompassing details such as the author’s name, year of publication, image type, dataset size, model architecture, accuracy, sensitivity, specificity, precision, F1-score, positive predictive values (PPV), negative predictive values (NPV), and area under the ROC curve (AUC).

Risk of bias assessment

In this umbrella review, the quality assessment of the included studies used the Joanna Briggs Institute (JBI) checklist, comprising 11 criteria designed to evaluate various aspects of an SR.

The set of eleven questions (Q1–Q11) was carefully designed to target specific domains, serving as a reliable measure of study quality with high sensitivity and specificity. These questions were used to evaluate the quality of the study across eleven distinct domains (D1–D11).

The response options for each criterion are “Yes,” “No,” “Unclear,” or “Not Applicable.” Two reviewers independently evaluated each study included in the review. Disagreements were resolved through further discussions between the reviewers to achieve consensus. The overall quality of each study was determined by counting the number of criteria that received a “Yes” response. To categorize the quality of these studies, the total possible scores, ranging from 0 to 11, were divided into three levels: poor quality (0–3 scores), medium quality (4–7 scores), and high quality (8–11 scores), aligning with categorization methods used in other research contexts. It is important to note that the JBI checklist itself does not prescribe a specific methodology for assessing study quality.

Screening of SRs and MAs

To perform a thorough literature search, we used the PubMed/Medline, Embase, Scopus, and Web of Science databases. Our initial search yielded 109 entries, which were carefully reviewed to remove duplicates. After this process, 55 studies remained for title and abstract review. Through detailed assessment, we identified 14 studies that met our inclusion criteria. After a thorough review of the full texts, we selected seven review articles to include in our comprehensive umbrella study (Prados-Privado et al., 2020; Khanagar et al., 2021; Musri et al., 2021; Khanagar et al., 2022; Mohammad-Rahimi et al., 2022; Singh & Raza, 2022; Talpur et al., 2022). Figure 1 shows a visual representation of the article selection process (adopted PRISMA flow diagram). Additionally, Table 2 lists the specific search terms used for each database and the corresponding number of results obtained from each.

Characteristics of the included SRs

One study that stood out for its comprehensive scope and high-quality reporting assessed by the JBI quality assessment checklist, was performed by Khanagar et al. (2022). The study investigated the effectiveness of AI-based models in identifying, diagnosing, and predicting DC. Out of 448 studies reviewed, 34 were included in the qualitative synthesis, using input datasets such as periapical radiographs, bitewing radiographs, panoramic radiographs, and other modalities to develop ANN and CNN models (Khanagar et al., 2022).

Despite differences between the included studies, all reviews used a similar approach and included various modalities (Prados-Privado et al., 2020; Khanagar et al., 2021; Musri et al., 2021; Mohammad-Rahimi et al., 2022; Singh & Raza, 2022; Talpur et al., 2022).

A notable point of contention across all included studies was the time limit for the inclusion of studies. While all reviews effectively screened and retrieved new articles based on their search periods, certain studies restricted their searches to articles indexed from specific years, such as 2010 (Mohammad-Rahimi et al., 2022), 2015 (Musri et al., 2021), and 2016 (Singh & Raza, 2022). The implications of these varying time limits warrant further discussion.

Regarding the overlapping included articles in these reviews, it should be noted that in accordance with the aim of this study, only studies that used “radiographic modalities input” were considered for the overlap assessment. The findings of this assessment are detailed in Table 5. Out of the 24 studies evaluated, only four were consistently included in multiple reviews. Specifically, the studies by Lee, Geetha, and Srivastav each appeared in four out of the seven reviews included in our current review. Meanwhile, the study performed by Choi, Eun & Kim (2016) was featured in three reviews.

In each review, various quality assessment tools were used. Three SRs used the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool guidelines (Khanagar et al., 2021, 2022; Mohammad-Rahimi et al., 2022), one followed Cochrane guidelines (Prados-Privado et al., 2020), and another used a self-created quality assessment checklist (Talpur et al., 2022). However, the quality assessment protocols and checklists used were inadequately described in the remaining two reviews (Musri et al., 2021; Singh & Raza, 2022).

Three of the included reviews adhered to the PRISMA–DTA guidelines as their reporting framework (Khanagar et al., 2021, 2022; Mohammad-Rahimi et al., 2022). Additionally, two other reviews followed the original PRISMA checklist for their reviews (Musri et al., 2021; Talpur et al., 2022). However, the remaining two reviews did not specify the guidelines or checklists used for their study reports (Prados-Privado et al., 2020; Singh & Raza, 2022).

Risk of bias and quality assessment of included studies

Three out of the seven included studies were classified as “high-quality” according to the JBI quality assessment checklist (Prados-Privado et al., 2020; Khanagar et al., 2022; Talpur et al., 2022). All remaining studies were categorized as “moderate-quality” (Figs. 2, 3) (Khanagar et al., 2021; Musri et al., 2021; Mohammad-Rahimi et al., 2022; Singh & Raza, 2022).

According to the JBI checklist, a main concern in SRs is the method of critical appraisal for included studies. To minimize bias and systematic errors in a systematic review, two or more reviewing team members must independently perform the critical evaluation of included studies, and this process should be duplicated. The review must clearly state that at least two reviewers independently performed the critical appraisal and consulted with each other as needed. This approach ensures that decisions regarding study quality and eligibility are based on a thorough and unbiased assessment (Aromataris et al., 2015). However, this issue is typically not comprehensively addressed in the text of several included reviews (Khanagar et al., 2021; Musri et al., 2021; Singh & Raza, 2022; Talpur et al., 2022).

A more critical issue in some included studies was the use of inadequate sources, which may have resulted in not covering all available evidence for the review (Musri et al., 2021; Singh & Raza, 2022). An SR needs to encompass all available evidence, which requires a thorough and comprehensive search strategy. This involves searching multiple electronic databases, including key bibliographic citation repositories. Additionally, a well-performed review should incorporate a search for grey literature or unpublished studies. This may involve exploring relevant websites or repositories of theses (Aromataris et al., 2015). Undertaking such extensive searches is crucial to minimize the risk of publication bias in an SR.

Evidence from high-quality reviews

In their 2022 study, Khanagar et al. (2022) aimed to assess the diagnostic accuracy and effectiveness of AI-based models in detecting, diagnosing, and predicting DC. After a thorough analysis, 34 articles meeting the selection criteria were critically reviewed. The study concluded that AI models demonstrate remarkable efficacy and are suitable for clinical application in identifying patients at higher risk of DC. These models also enhance diagnostic accuracy, treatment quality, and patient outcomes. The predictive capabilities of these models are valuable for developing preventive dental care strategies and personalized oral hygiene and dietary plans for individuals prone to DC. Despite their impressive performance, the study identified limitations in the size and diversity of the datasets used in the articles. Therefore, further training and validation of these models are necessary to enhance their effectiveness (Khanagar et al., 2022). It is important to note that this study encompassed various modalities for model development, including tabular, 2D, and 3D data. The conclusions drawn from different modalities may need modification when evaluating a certain modality, such as radiographs. Ultimately, eighteen out of the 34 studies included in Khanagar et al.’s (2022) review were selected for further evaluation in accordance with the aims of this overview.

In 2020, Prados-Privado et al. (2020) reviewed the latest advancements in neural networks for caries detection and diagnosis. This review encompassed thirteen studies using techniques such as near-infrared light transillumination and periapical and bitewing radiography. The studies included varied image databases, ranging from 87 to 3,000 images, with an average of 669 images per study. Seven studies involved expert dentists who annotated the presence of DC in each image. Not all studies provided a detailed definition of caries or specified the type of carious lesion detected. The review highlighted the variability in diagnostic efficacy among AI models using different algorithms, emphasizing the need for verification of their generalizability and reliability. Before these models can be effectively implemented in clinical practice, a comparative analysis of the results from each algorithm is necessary (Prados-Privado et al., 2020). Four studies from this review were selected for further evaluation in this overview.

In their 2022 study, Talpur et al. (2022) performed an SR to investigate the correlation between DC and ML. They identified 133 articles and eventually included twelve in their review. The authors evaluated the application of DL in dental imaging for diagnosing DC, specifically proximal, occlusal, and root caries. The SR concluded that these twelve studies provided substantial evidence in distinguishing dental X-ray images with and without caries. Among these algorithms, the neural network backpropagation algorithm was identified as the most effective, achieving up to 99% accuracy in detecting DC (Talpur et al., 2022). The study emphasizes the need for further extensive research to diagnose various types of DC, taking into account disease progression (chronic, acute, and arrested), which could offer insights into the severity and extent of caries. Additionally, four studies from this review were chosen for detailed analysis.

Evidence from moderate-quality reviews

In 2020, Khanagar et al. (2021) performed a study to evaluate the development and efficacy of AI applications in dentistry, focusing on their role in diagnosis, clinical decision-making, and prognosis prediction. Following specific inclusion and exclusion criteria, they selected 43 articles for review. The review found that most studies used AI models based on CNNs and artificial neural networks (ANNs). These models have demonstrated the ability to achieve or exceed the precision and accuracy of trained dental specialists. The authors suggested that AI could serve as a complementary tool for dentists, enhancing diagnostic accuracy, assisting in treatment planning, and improving outcome predictions. Furthermore, AI and DL systems could provide diagnostic support to general dentists, thereby saving time and increasing efficiency. The use of AI to provide second opinions and improve diagnostic accuracy was also highlighted (Khanagar et al., 2021).

Mohammad-Rahimi et al. (2022) performed an SR to evaluate DL studies focused on caries detection. They included studies that used DL models with various dental imaging modalities, such as radiographs, photographs, and optical coherence tomography images, published after 2010. Out of 252 potential references, 48 studies were fully reviewed, with 42 ultimately included in their analysis. The accuracy of caries detection models varied, achieving 82–99.2% on periapical radiographs, 87.6–95.4% on bitewing radiographs, and 86.1–96.1% on panoramic radiographs. While the accuracy was promising, they noted that the overall quality and reporting of the studies were generally low. They concluded that DL models could aid in decision-making regarding the presence or absence of caries (Mohammad-Rahimi et al., 2022). Five studies from this review were selected for further evaluation in this overview.

Musri et al. (2021) focused on evaluating DL and CNNs for detecting and diagnosing early-stage DC using periapical radiographs. Their research, encompassing evidence from 2015 to 2021, revealed that deep CNN-based detectors effectively identified caries, enabling practitioners to discern changes in the location and morphology of DC lesions. Despite generally small datasets used in the studies reviewed, both traditional and optimized DL methods were used, although the studies did not differentiate between early-stage and root caries. The authors noted that DL algorithms, with their complex and evolving layers, continue to advance, significantly enhancing their accuracy in object detection and segmentation. They concluded that CNN algorithms provide thorough, reliable, and accurate assessments of images and disease detection, thereby improving the efficiency and effectiveness of diagnosis and prognosis of DC in periapical radiographs (Musri et al., 2021).

Singh & Raza (2022) performed a study to explore recent advancements in DL-based analysis of dental and maxillofacial images, reviewing literature from 2016 to 2020. They included 75 articles that applied DL to dental image classification, segmentation, object detection, and other image-processing activities. The authors observed that AI and neural network applications in dental image analysis have demonstrated superior performance, occasionally surpassing that of dental experts. However, they highlighted the importance of validating the accuracy of these systems across various imaging devices and modalities to standardize data. Given the prevalence of conditions such as periodontally compromised teeth, tooth caries, and apical lesions, they suggested that compiling large datasets of similar conditions could help achieve higher levels of accuracy more efficiently (Singh & Raza, 2022).

Limitations

The current models exhibit strong performance, but they are constrained by limitations related to the size and diversity of the datasets used in the studies. Consequently, further training and validation are essential to optimize their performance. This study provides a thorough and systematic evaluation of DL in caries detection, facilitating a thorough synthesis and comparison of the results. It is important to note that our search was limited to a specific time period and scope, focusing solely on DL while excluding alternative image analysis methods previously used. These constraints were justified by the substantial and diverse evidence we obtained. Despite our intention to conduct an MA, the extensive variability and inadequate reporting quality in the studies limited our ability to gather the necessary details for such an analysis.

Strengths

This is the first overview in this field aimed at identifying knowledge gaps and evaluating the quality of SR studies performed in this context. Our findings suggest that CNN algorithms provide thorough, reliable, and accurate image analysis and disease detection. CNN algorithms facilitate effective and efficient diagnosis and improving the prognosis of DC in periapical radiographs.

Future directions

The studies referenced in the SRs used relatively small datasets for traditional and advanced DL methods without distinguishing between early and root caries. There is a need for accurate detection and diagnosis of DC to reduce the costs associated with oral health management and enhance the likelihood of preserving natural teeth. Many researchers advocate for the development of a large-scale, publicly accessible dataset to achieve clinically reliable accuracy. It is also important to focus on automating preprocessing tasks using unsupervised or semi-supervised training to reduce errors from manual procedures.

Despite the promising advancements in the application of deep learning (DL) and convolutional neural networks (CNN) for detecting dental caries (DC) using radiographic images, several limitations persist that require attention from future research efforts.

Weaknesses in existing methods

Dataset size and diversity: Many current studies rely on relatively small and homogeneous datasets, which limits the generalizability and robustness of the models. These datasets often lack diversity in terms of demographic variables, imaging modalities, and the extent of disease progression. Consequently, the models trained on such datasets may perform well in controlled environments but struggle to maintain accuracy when applied to broader, real-world scenarios.

Overfitting issues: The risk of overfitting remains a significant concern, particularly when models are trained on small datasets with insufficient variability. Overfitting occurs when a model learns to perform well on the training data but fails to generalize to new, unseen data. This compromises the model’s applicability in clinical settings.

Lack of standardization: There is a notable lack of standardization in the training and validation processes across different studies. Variations in the types of radiographic images, annotation protocols, and quality assessment methods hinder the ability to compare results directly and derive consistent conclusions.

Insufficient attention to clinical relevance: While many DL models achieve high accuracy rates, there is often insufficient focus on ensuring that these models are clinically relevant and actionable. Models may detect lesions with high sensitivity, but without considering the clinical decision-making process, such detections may not translate into improved patient outcomes.

New directions for future research

Development of large-scale, diverse datasets: Future research should prioritize the creation and utilization of large, diverse, and publicly accessible datasets. These datasets should encompass a wide range of patient demographics, imaging modalities, and varying degrees of caries severity. This will improve the generalizability and robustness of AI models, making them more reliable in diverse clinical settings.

Focus on unsupervised and semi-supervised learning: To reduce the reliance on large labeled datasets and minimize manual errors, researchers should explore unsupervised or semi-supervised learning techniques. These approaches can leverage unlabeled data, which is abundant, to improve model training and performance without extensive manual annotation.

Integration of multi-modal data: Combining data from various imaging techniques (e.g., periapical radiographs, bitewing radiographs, optical coherence tomography) could enhance the accuracy of DL models. Future studies should investigate how integrating multi-modal data can improve diagnostic accuracy and provide a more comprehensive assessment of dental caries.

Improvement in interpretability and explainability: As AI models become more complex, it is crucial to develop methods that improve the interpretability and explainability of these models. Clinicians need to understand how AI algorithms arrive at their conclusions to trust and effectively use these tools in practice. Research should focus on creating models that provide clear, understandable outputs that align with clinical reasoning.

Personalized dental care algorithms: Future work should aim to develop AI models that can contribute to personalized dental care. This involves creating algorithms that not only detect caries but also predict disease progression, suggest tailored treatment plans, and provide patient-specific preventive measures based on individual risk factors.

Validation and benchmarking across multiple sites: It is essential to validate AI models across multiple clinical settings and institutions to ensure their generalizability and reliability. Future research should focus on cross-institutional studies that benchmark the performance of DL algorithms, using standardized protocols and evaluation metrics.

Exploration of new algorithms and architectures: Continued exploration of novel DL architectures, such as transformers and attention-based models, may yield further improvements in caries detection and diagnosis. These advanced architectures could offer better performance in handling complex patterns in medical imaging data, providing more accurate and reliable diagnostics.