Integrating whole genome sequencing and machine learning for predicting antimicrobial resistance in critical pathogens: a systematic review of antimicrobial susceptibility tests

Introduction

Antimicrobial resistance (AMR) is the ability of bacteria to withstand antimicrobial treatments, particularly antibiotics. Infections caused by antibiotic-resistant bacteria are a significant concern for modern healthcare, posing a serious public health risk (Ahmad et al., 2023). Projections estimate that bacterial infections could result in approximately 10 million deaths annually by 2050 (Lüftinger et al., 2023). A recent meta-analysis on the impact of resistant bacteria on human health revealed that in 2019, antibiotic-resistant bacteria (ARBs) directly caused 1.27 million deaths, with an additional 4.95 million deaths associated with ARBs (Antimicrobial Resistance Collaborators, 2022). Moreover, ARBs are identified as a leading cause of mortality in low-income countries (Antimicrobial Resistance Collaborators, 2022; Ruiz-Blanco et al., 2022). Infections with Escherichia coli, Staphylococcus aureus, Klebsiella pneumoniae, Acinetobacter baumannii, and Pseudomonas aeruginosa, all classified as critical and high-priority pathogens (CP) by the World Health Organization (WHO) (Ahmad et al., 2023; Antimicrobial Resistance Collaborators, 2022), account for a significant portion of these deaths. In 2019, over 100,000 deaths were attributed to antimicrobial resistance (AMR) caused by a single pathogen-drug combination: methicillin-resistant S. aureus (MRSA). Six other combinations caused 50,000 to 100,000 deaths each, including multidrug-resistant E. coli, fluoroquinolone-resistant E. coli, carbapenem-resistant A. baumannii, carbapenem-resistant K. pneumoniae, and third-generation cephalosporin-resistant K. pneumoniae. Additionally, a recent assessment of the clinical pipeline revealed the development of 50 antibiotics, but only 12 showed efficacy against certain priority Gram-negative bacteria (Butler & Paterson, 2020; Pormohammad, Nasiri & Azimi, 2019).

Research indicates that the rapid administration of appropriate antimicrobials significantly improves patient outcomes. For instance, in cases of bacteremia, the risk of death doubles if effective antibiotics are not administered within 24 h. Globally, only about half of antibiotic prescriptions are accurate. Consequently, quick point-of-care diagnostic tests are crucial for addressing this issue (Ahmad et al., 2023; Milani et al., 2019).

The current culture-based methods for detecting and diagnosing pathogenic diseases are insufficient. Most culturable bacteria associated with diseases can be detected after 24-48 h of incubation. Additionally, pathogen identification often requires an extra 2-4 h, and if AMR is suspected, antibiotic susceptibility testing (AST) adds another 18-24 h. Consequently, the total time to collect patient samples and obtain information on antibiotic susceptibility patterns in clinical practice can range from 2 to 4 days at best (Ahmad et al., 2023; Taxt et al., 2020).

Emerging micro- and nanotechnologies for bacterial identification and AST include phenotypic methods like microfluidic-based bacterial culture, and molecular techniques such as multiplex PCR, hybridization probes, nanoparticles, synthetic biology, and mass spectrometry. While PCR and mass spectrometry have improved bacterial detection in positive cultures, they have limitations. PCR requires a predetermined target, and MALDI-TOF mass spectrometry is costly (Ahmad et al., 2023; Taxt et al., 2020; Humphries et al., 2023). Additionally, these methods, which assess one or more resistance genes, are inadequate for predicting antimicrobial susceptibility because resistance often results from a complex interplay of resistance genes, regulatory factors, and mutations that together produce a phenotypic susceptibility profile (Humphries et al., 2023).

Whole genome sequencing (WGS) offers a solution to some of these challenges by obviating the requirement for specialized primers or probes. Additionally, with the increasing affordability of real-time sequencing, WGS has emerged as a feasible alternative to the laborious, culture-dependent methods of the past (Ahmad et al., 2023). Moreover, genome sequencing data offer an additional dimension to AMR research, enabling the analysis of genetic pathways underlying AMR in individual strains (Su, Satola & Read, 2019; VanOeffelen et al., 2021). Several publicly accessible services have been established to assist in identifying AMR indications based on the presence of resistance-associated single nucleotide polymorphisms (SNPs) and genes (VanOeffelen et al., 2021; Bortolaia et al., 2020). Integration of AST data with genome sequences also holds promise in uncovering genomic regions directly involved in resistance, influenced by epistasis, or linked to the emergence of AMR (VanOeffelen et al., 2021; Hendriksen et al., 2019).

The utilization of machine learning (ML) methods for forecasting AMR indicators and pinpointing genetic regions associated with resistance has garnered considerable interest in recent literature (VanOeffelen et al., 2021; Anahtar, Yang & Kanjilal, 2021). ML technologies leverage a wide array of variables inherent in genomic data to construct nonlinear models that forecast phenotypic AST outcomes (Humphries et al., 2023). ML offers an alternative method for predicting AMR from sequence data without necessitating prior knowledge of chromosomal alterations or mobilizable genes (Aytan-Aktug et al., 2020; Moradigaravand et al., 2018a). Numerous ML approaches can consider the implications of multiple mutations and/or mobilizable genes. Various studies have employed different ML techniques to forecast AMR profiles for diverse bacterial species and drug combinations (Aytan-Aktug et al., 2020; Moradigaravand et al., 2018a; Drouin et al., 2019). The primary distinction among these studies lies in how bacterial genomes are transformed into features, which are subsequently input into ML algorithms (Aytan-Aktug et al., 2020). Unlike culture-based AST or nucleic acid amplification tests, which are frequently constrained in the scope of resistant phenotypes ascertainable through a single test, WGS-AST enables the simultaneous determination of antibiotic resistance phenotypes across the entirety of the genome. Furthermore, it facilitates the screening of phenotypes influenced by multiple loci with ease (Ahmad et al., 2023; Lüftinger et al., 2023; Humphries et al., 2023; Su, Satola & Read, 2019; VanOeffelen et al., 2021). However, no systematic review has been identified that has evaluated the ability of ML to predict antimicrobial resistance in CP using WGS. Therefore, the objective of this systematic review is to evaluate the efficacy of ML approaches in predicting AMR in CP, considering WGS-AST.

Materials and Methods

Results

Study selection

After searching as indicated, 773 studies were recognized in electronic databases. After subtracting duplicates and employing suitability conditions, 68 documents experienced a thorough full-text review. Omissions through this assessment were primarily due to the absence of WGS and AST of CP, or to inadequate data in the model validation procedure. Following the last phase of the suitability valuation, this study finally incorporated 26 articles. Figure 1 illustrates a detailed depiction of the examination flowchart.

Discussion

This study was conducted to evaluate ML predictions for AMR in CP utilizing WGS and AST was performed. Although LR was usually used for prediction, RF and XGBoost were also commonly utilized. Notably, RF demonstrated the highest AUC values compared to LR. Furthermore, other algorithms such as SVM, AdaBoost, and Neural Networks were utilized. Importantly, all ML models accurately predicted resistance patterns in CP across multiple antibiotics using WGS and AST.

To the best of our knowledge, this is the inaugural systematic review to evaluate the efficacy of ML models in predicting AMR utilizing WGS and AST specifically for critical and high-priority pathogens. This inquiry builds upon a previous systematic study proposing ML as a promising tool for AMR prediction (Tang et al., 2022). Alarmingly, nearly half of the research included in that publication did not delineate resistance patterns, whereas all studies reviewed herein did so. The integration of WGS and AST data into our investigation is imperative for enhancing the robustness and practicality of ML models in AMR prediction. By elucidating resistance patterns, our review elucidates the efficacy of these models in guiding antimicrobial therapy.

WGS data provide an alternative perspective on AMR, enabling researchers to assess the genetic pathways that confer AMR in each strain (Ahmad et al., 2023; Lüftinger et al., 2023; Humphries et al., 2023; Noman et al., 2023; Yang & Wu, 2022; Stracy et al., 2022; Pearcy et al., 2021; Benkwitz-Bedford et al., 2021; Khaledi et al., 2020; Hyun et al., 2020; Pataki et al., 2020; Macesic et al., 2020; Kim et al., 2020). Numerous publicly accessible resources have been developed to assist in identifying AMR indicators by detecting the presence of genes and single nucleotide polymorphisms that confer resistance (Su, Satola & Read, 2019; VanOeffelen et al., 2021). The combination of AST data with WGS also has the potential to unveil genomic regions directly involved in resistance, altered due to epistasis, or linked to the occurrence of AMR (VanOeffelen et al., 2021; Hendriksen et al., 2019). Several resources, such as the National Center for Biotechnology Information, European Molecular Biology Laboratory- European Bioinformatics Institute, Relational Sequencing TB Data Platform, AR Isolate Bank, and Pathogenwatch, offer genome datasets matched with AST data for further analyses like comparative genomics and modeling (VanOeffelen et al., 2021; Sayers et al., 2020; Matamoros et al., 2020).

While WGS provides a valuable genetic blueprint that can predict AMR, integrating AST data enhances the ability to confirm phenotypic resistance patterns. This approach addresses the inherent limitation that genomic data alone cannot fully account for all phenotypic expressions of resistance. The combination of WGS and AST data not only supports the prediction of resistance phenotypes but also helps to uncover genetic interactions, such as epistasis, that may influence AMR (VanOeffelen et al., 2021; Hendriksen et al., 2019). By employing a range of ML models, including ensemble methods like RF and XGBoost, we can better capture the complexity of AMR prediction, making these models more applicable to clinical practice.

In clinical settings, the implementation of such integrated approaches can improve the accuracy of AMR predictions, thereby guiding more effective antimicrobial therapy. We recommend that future studies continue to explore the synergistic use of WGS and AST data, alongside the development of more sophisticated ML models, to further refine the predictive power and clinical utility of these methods.

Several studies have underscored the importance of providing information about AMR testing. Without AST information, the AUC values ranged from 0.73 to 0.79. Nonetheless, incorporating AST led to even higher AUC scores, which ranged from 0.80 to 0.88 (Lewin-Epstein et al., 2021). Optimization replications indicate that, notwithstanding diffident AUC values, antibiotic selection guided by personalized antibiograms can equal or surpass physician achievement. Furthermore, such selection yielded coverage rates akin to those observed in real-world scenarios, while requiring fewer broad-spectrum antibiotics (Corbin et al., 2022). This underscores a persistent and critical challenge in antibiotic stewardship.

Likewise, it has been verified that the quality of initial data and the precision of metagenomic binning are crucial for the effectiveness of subsequent applications like genomic AST. A workflow designed for native samples with low bacterial complexity and adequate on-target sequencing depth demonstrates comparable performance to genomic AST on isolate sequencing data (Lüftinger et al., 2021).

The AUC serves as a widely adopted standard measure for assessing model functioning. This measured was identified as the main success indicator in both our study and a previous review on AMR. Nevertheless, the earlier evaluation did not encompass AST in all the scrutinized publications, nor did it evaluate WGS (Tang et al., 2022). Notably, there is a disparity in the range of AUC values between the two studies for LR outcomes (0.76−0.96 in our assessment versus 0.50−0.83), as well as for other ML results (0.48−0.92 versus 0.83−0.91 for XGBoost and 0.66−0.97 for RF in our study). The differences in selection criteria between the two assessments present challenges in comparing the outcomes directly. Nevertheless, it is conceivable that the incorporation of WGS-AST impacts the outcomes, and that the models react differently based on the input factors. In this framework, research has demonstrated that while comparing results across different settings has its limitations, some models established previously (Mintz, Chowers & Obolski, 2023) exhibit superior performance compared to other studies (Yelin et al., 2019; Feretzakis et al., 2020). Curiously, these ML models demonstrated good performance on a diverse dataset, which included various microorganisms, test informants, and clinical divisions (Mintz, Chowers & Obolski, 2023). A previous study (Feretzakis et al., 2020) predicted AMR using data from a specific medical unit, based on the Gram stain result of the sample, achieving an AUC of 0.72. In contrast, another study (Yelin et al., 2019) focused on predicting AMR exclusively in cases using urine samples and limited the analysis to three microorganisms, achieving an AUC of 0.83.

Typically, predictors are selected by means of both unadjusted and multivariate LR models. Here, usual input risk features contain AMR patterns, WGS, colonization, ART, and past AMR circumstances. These characteristics are narrowly associated with AMR and can be utilized as predictors in various ML models and risk score assessments (Goodman et al., 2016). However, it is challenging to determine if including additional variables, such as underlying disorders, improves prediction exactitude. Besides, well-known issues, such as the practice of proton pump inhibitors (PPIs) (Shang, Lin & Goetz, 2000), can be ignored in some studies. Consequently, further prospective research is needed to better understand the impact of PPI usage.

Another method for validating final predictors is to use feature selection processes (Tang et al., 2022; Mintz, Chowers & Obolski, 2023; Yelin et al., 2019; Feretzakis et al., 2020). While predictors identified by these algorithms align with those proposed by LR models or previous data, others, such as admission times, have ambiguous relationships with AMR. Domain expertise and a structured approach are considered essential for sorting through the substantial quantities of data from health organizations (Tang et al., 2022).

The current study’s findings suggest that a ML forecast based on WGS-AST could aid in guiding antibiotic recommendations for confirmed carbapenemase-producing CP infections. A previous systematic review (Tang et al., 2022) reported similar results. However, other studies have compared the efficacy of ML systems to risk scores, with inconsistent outcomes (Moran et al., 2020; Lee et al., 2021). Indeed, the findings in this field vary significantly. One comprehensive review, which aimed to develop diagnostic or prognostic clinical prediction models for binary outcomes using clinical data, found no evidence that ML outperformed LR, contradicting the results of two other systematic reviews. One review (Beunza et al., 2019) indicated that ML algorithms can enhance the diagnostic and prognostic capabilities of traditional regression techniques, while another (Sufriyana et al., 2020) recommended reanalyzing existing LR models for various outcomes and comparing them to algorithms adhering to established standards. Although risk scores can provide valuable bedside assistance, it is assumed that health organisms integrated with ML may address this concern by leveraging considerable volumes of information (Tang et al., 2022). The primary advantage of ML lies in its continuous learning development, leading to superior model exactitude and a broad range of uses in healthcare information. Dissimilar to conventional statistical approaches, ML does not rely on specific assumptions, which are frequently overlooked or critically examined in clinical information (Rajula et al., 2020). Consequently, the choice of algorithms would be directed by the investigation topic and the purpose framework.

Partial information is unavoidable in certain studies, leading to statistical difficulty and bias in ML projections. Another challenge is data disparity in the AMR prediction model (Tang et al., 2022). This discrepancy adversely affects calculation functioning, as classifiers incline to favor the majority class to diminish global inaccuracy proportions (Japkowicz, 2000). To alleviate this subject, methods such as resampling, correcting hyperparameters, and meticulous method choice may be used (Tang et al., 2022). Upcoming investigators must cooperate with distinct groups to perform high-grade models.

While classification-based ML models have been pivotal in predicting AMR phenotypes by categorizing pathogens as resistant or susceptible, it is essential to consider the regression-based approaches that directly predict the minimum inhibitory concentration (MIC) (Yang, Su & Wu, 2023). MIC is a critical quantitative measure that reflects the lowest concentration of an antibiotic required to inhibit bacterial growth. Classification models typically rely on established MIC breakpoints to determine resistance or susceptibility, as defined by standards from organizations such as the Clinical and Laboratory Standards Institute (CLSI) (Noman et al., 2023; Benkwitz-Bedford et al., 2021). However, these breakpoints are periodically reviewed and updated, which could affect the consistency of classification-based predictions. In contrast, regression models that predict MIC values provide a more detailed and adaptable understanding of the bacterial response to antibiotics, allowing for more precise phenotypic annotations (Yang, Su & Wu, 2023).

Relevant studies included in this systematic review underscore the importance of regression-based machine learning models in the accurate prediction of MIC values (Noman et al., 2023; Benkwitz-Bedford et al., 2021; Khaledi et al., 2020; Hyun et al., 2020; Pataki et al., 2020; Macesic et al., 2020; Nguyen et al., 2018). Nguyen et al. (2018) utilized WGSD to predict MICs across various bacterial species, demonstrating the potential of these models to enhance the understanding of resistance mechanisms at a quantitative level. Similarly, Pataki et al. (2020) employed ML models to predict MIC values, which enabled a more nuanced interpretation of antimicrobial resistance that goes beyond binary classification. Yang, Su & Wu (2023) further expanded on this approach by applying a pan-genome-based feature selection method to improve the accuracy of MIC predictions in Salmonella enterica. Their work highlighted that the selected genomic features, including novel genes not previously associated with AMR, contributed significantly to the accurate prediction of MIC. This suggests that regression-based models can uncover new genetic determinants of resistance, providing insights that are not easily captured by classification methods alone. Incorporating MIC prediction into AMR research thus offers a robust complement to classification-based approaches, enhancing the granularity and applicability of machine learning in clinical microbiology.

There is no well-established tool for evaluating bias risk in ML prediction studies. One study (Delpino et al., 2022) utilized the TRIPOD statement to characterize study quality, while other studies (Fleuren et al., 2020; Christodoulou et al., 2019) employed the QUADAS-2 criteria. The TRIPOD statement serves more as a checklist than a bias assessment tool, whereas the QUADAS-2 criteria are widely used to evaluate the quality of diagnostic accuracy studies (Whiting et al., 2011). PROBAST (Moons et al., 2019) has also been used to evaluate ML in predicting AMR, as demonstrated in Tang et al. (2022).

The present review is subject to limitations. The studies under review exhibited considerable variability stemming from variations in outcomes, predictors, ML, and hyperparameters, among other factors. Most of the studies included in the review were identified as having a substantial risk of bias, precluding the possibility of conducting a meta-analysis. Two systematic reviews focusing on ML-based prediction models noted notable disparities among the studies they evaluated (Tang et al., 2022; Fleuren et al., 2020). One of these reviews observed a heterogeneity exceeding 97% (Tang et al., 2022). Another study discovered that there was no discernible discrepancy in AUC between ML and LR across 145 assessments exhibiting a low risk of bias (0.00, 95% CI −0.18 to 0.18). Nevertheless, in 137 instances characterized by a high risk of bias, ML exhibited a substantially superior AUC of 0.34 (0.20−0.47) (Christodoulou et al., 2019). Cochrane advises exercising caution when interpreting data with an I² value exceeding 50%, as it signifies considerable heterogeneity (Higgins et al., 2003; Schroll, Moustgaard & Gøtzsche, 2011).

While our study focused on a subset of critical and high-priority pathogens identified by the WHO, including E. coli, S. aureus, K. pneumoniae, A. baumannii, and P. aeruginosa, we acknowledge that other important microorganisms were not included. Notably, Salmonella spp., as well as other WHO-designated high-priority pathogens such as Enterococcus faecium, Helicobacter pylori, and Campylobacter spp., were not part of our analysis. A systematic review that encompasses these additional microorganisms would provide a more comprehensive understanding of the global burden of antimicrobial resistance. This would be particularly important for informing public health policies and interventions aimed at reducing the transmission of AMR through various routes, including food and water. Furthermore, such a review would help identify knowledge gaps and research priorities for addressing AMR in a broader range of microorganisms. Future studies should consider including these microorganisms to provide a more complete picture of the global AMR landscape.

Conclusions

By integrating whole genome sequencing and antimicrobial susceptibility testing data in critical high priority pathogens, machine learning demonstrates significant potential for predicting antimicrobial resistance. Machine learning models, particularly Random Forest, XGBoost, and logistic regression, offer valuable clinical decision support by accurately predicting antimicrobial resistance in critical pathogens. This can assist healthcare providers in making informed treatment decisions, optimizing antibiotic use, and improving patient outcomes. Standardized guidelines are imperative to uphold consistency in forthcoming studies.

Supplemental Information