Evaluating the efficacy of AI-driven intrusion detection systems in IoT: a review of performance metrics and cybersecurity threats

Search strategy

The search strategy involved systematic queries across two major academic databases Scopus, and Web of Science. Both databases were selected due to their extensive repositories of scholarly literature, which support a thorough exploration of the research domain. The stringent indexing standards applied by these databases ensure the academic credibility and reliability of the included studies (Martín-Martín et al., 2018; Pranckutė, 2021). Keywords such as (“artificial intelligence” OR AI) AND (“intrusion detection system” OR IDS) AND (“internet of things” OR IoT) AND (efficacy OR performance OR accuracy OR precision OR recall OR “detection rate”) AND (“cybersecurity threat” OR “security attack” OR “cyber-attack” OR “cyber threats”) were used in various combinations. The search was conducted in April 2024 to capture the most current and pertinent literature available at that time.

Study inclusion and exclusion

Strict inclusion and exclusion criteria were employed to guarantee that only highly relevant studies were incorporated into this meta-analysis. The selection parameters used in this meta-analysis, as detailed in Table 2, were designed to enhance the accuracy, objectivity, and practical relevance of the findings by focusing specifically on research related to AI-based intrusion detection systems in IoT.

Study selection

The study selection process followed the PRISMA 2020 guidelines to maintain rigor and transparency in identifying pertinent research on the efficacy of AI-driven IDS in IoT environments. During the identification phase, a comprehensive search of two major databases yielded a total of 203 records, with 153 from Scopus and 50 from Web of Science. In the screening phase, 88 records were excluded due to irrelevance, and nine duplicate records were identified and removed, leaving 79 records for full-text screening. During the eligibility assessment phase, 17 records were excluded for not meeting the basic inclusion criteria, and seven reports could not be retrieved. Of the 55 reports assessed for eligibility, four studies were excluded for not applying AI techniques. Ultimately, 51 studies were included in the final meta-analysis.

Eligibility criteria (PICO)

The inclusion criteria were defined using the Population, Intervention, Comparison, Outcomes (PICO) framework, studies were included if they: (1) were published between 2016 to 2025, (2) were written in English, and (3) focused primarily on AI-based IDS techniques in IoT environments. Studies were excluded if they: (1) focused on unrelated technologies (e.g., non-AI approaches or non-IoT contexts), or (2) lacked empirical or experimental data, theoretical articles, non-English articles, dissertations, and studies without performance metrics. This ensured relevance, methodological rigor, and comparability for the meta-analysis. Table 3 outlines the specific PICO elements applied in determining study eligibility.

Data extraction

We used a structured Microsoft Excel spreadsheet to systematically extract key information from each study included in the review. The extracted data included the authors, year of publication, study design, IoT application domain, types of AI models used, feature engineering techniques, datasets utilized, and performance metrics such as accuracy, precision, recall, and F1-score. Additional details captured comprised threat scenarios addressed, adversarial strategies applied, comparisons with conventional methods, and the country of the study. Data extraction was conducted independently by two reviewers, with any disagreements resolved through discussion to ensure accuracy and reduce bias.

To handle variations in reported metrics, we applied a data harmonization strategy. Accuracy was chosen as the primary metric for quantitative comparison due to its consistent reporting across studies. Supplementary metrics such as F1-score, precision, and recall were analyzed when available. Studies that lacked at least one standard performance metric were included in the qualitative synthesis but excluded from the quantitative analysis. To account for variations in datasets and AI methodologies, we categorized studies by AI type and IoT application domain. This classification allowed for more meaningful subgroup comparisons rather than attempting a single aggregated analysis.

Quality assessment

To maintain methodological rigor, we assessed the quality of each included study using a modified version of the Critical Appraisal Skills Programme (CASP) checklist (Long, French & Brooks, 2020). Evaluation criteria covered the clarity of research objectives, validity of datasets, completeness of performance metrics, soundness of experimental design, and relevance to real-world IoT environments. Studies were rated on a standardized scale, and only those meeting a predefined quality benchmark were included in the meta-analysis. This thorough vetting process ensured that only high-quality, credible studies informed our findings.

Model taxonomy and classification

To maintain uniform terminology throughout the meta-analysis, a standardized taxonomy is established for classifying Intrusion Detection System (IDS) models. This classification differentiates between deep learning, ensemble, and hybrid approaches based on their structural design and methodological integration. The framework is applied consistently across the manuscript to ensure clarity and prevent misinterpretation Table 4.

Study characteristics

The section discusses the demographic characteristics of the included studies based on annual publications and countries of the published included studies. Table 5 provides a comprehensive summary of the 51 studies analyzed in this study, outlining the AI techniques employed, datasets utilized, and the specific cybersecurity threats addressed. This overview allows readers to discern recurring methodological trends and differences among the studies. Commonly used public datasets included CICIDS 2017 (Indra et al., 2024), UNSW-NB15 (Keshk et al., 2023; Mousavi, Sadeghi & Sirjani, 2023), and NSL-KDD (Sadhwani et al., 2025; Tawfik, 2024), while some research drew on proprietary or simulated IoT data. The primary focus areas were intrusion detection, DDoS attacks, and malware classification, highlighting prevailing concerns in AI-driven smart grid and IoT security. This table lays the groundwork for the detailed qualitative and quantitative analyses presented in the subsequent sections.

Annual publications

The included studies are distributed across several years, with the highest number of studies occurring in 2024, where 20 studies were published. Following closely are 2023 and 2025, with 12 and 11 studies, respectively, reflecting ongoing research activity in recent years. The year 2022 also saw significant contributions, with six studies published. Fewer studies were recorded in 2018 and 2020, each with 1 study. This shows a clear increase in research output starting from 2022, peaking in 2024, and continuing strongly into 2025 as shown in Fig. 2.

Countries of publications

The included studies are primarily from India, which stands out with 11 studies, making it the leading contributor. Saudi Arabia follows with seven studies, showing its strong involvement in research on IoT and cybersecurity. China is also well-represented with four studies, reflecting its active role in AI and intrusion detection research. Other countries like Jordan, Algeria, and Australia each contribute three studies, highlighting their regional importance in this area. Egypt, France, the USA, and Morocco each appear twice, suggesting a significant interest in the field from these nations. Countries such as Iran, the United States, the United Kingdom, Nigeria, Bangladesh, Bulgaria, Malaysia, Yemen, the UAE, Pakistan, Thailand, and Italy each have one study, showcasing the global scope of research in AI-driven intrusion detection systems as shown in Fig. 3. Among the 51 studies reviewed, a clear geographic concentration was observed with the majority originating from India, Saudi Arabia, and China. This prominence is largely due to these nations’ significant research productivity in IoT-enabled smart grids and AI applications driven by robust national investments and access to extensive empirical data from major smart infrastructure initiatives. Research contributions from other regions were relatively sparse, indicating the existing global distribution of scholarly activity rather than any selection bias in our review process.

Comparative analysis of AI models’ performance metrics in IoT-based intrusion detection

As reported in Table 6, a broad range of AI models including machine learning, deep learning, and hybrid approaches have been applied to intrusion detection in IoT environments, exhibiting significant variation in their reported performance metrics. While some studies provided comprehensive evaluations including accuracy, precision, and F1-score, others omitted one or more of these key indicators. For instance, Saheed, Omole & Sabit (2025) and Rasheed & Alnabhan (2024) demonstrated near-perfect accuracy, precision, and F1 scores using models like long short-term memory (LSTM) and CatBoost. Some studies like Gueye et al. (2023), Sneha & Prasad (2024) and Kulrujiphat & Kulrujiphat (2024) reported only general high accuracy without specifics, while others like Tyagi et al. (2024) and Almiani et al. (2020) did not provide any quantitative performance metrics. Notably, Keshk et al. (2023) and Rehman et al. (2025) were among the few that provided a comprehensive set of metrics across multiple datasets. These variations highlight the challenge of direct comparison but also underscore a consistent trend of AI models achieving strong performance in smart grid intrusion detection, even when one or more metrics were not reported. The overall trend highlights the growing efficacy of AI-driven systems, particularly ensemble and hybrid models, in addressing complex cybersecurity threats within IoT networks.

Publication bias

The contour-enhanced funnel plot in Fig. 4 provides a visual assessment of potential publication bias in the meta-analysis evaluating the effectiveness of AI-driven IDS in IoT environments. Created using R version 4.4.3, the plot displays individual studies by their effect sizes (x-axis) and standard errors (y-axis), where studies with greater precision (smaller standard errors) are positioned higher, and those with less precision appear lower. In a scenario free from publication bias data points would be expected to distribute symmetrically around the central vertical line representing the pooled effect estimate forming a characteristic inverted funnel. In this case, while the plot retains a rough funnel shape a clear asymmetry is evident particularly a concentration of studies on the right side and a relative absence on the left especially in zones indicating statistically significant negative effects. This skew implies a potential bias toward publishing studies with favorable or positive outcomes for AI-based IDS models, while those reporting null or negative effects appear underrepresented. The shaded regions within the funnel reflect different levels of statistical significance with lighter zones indicating highly significant findings. A substantial number of studies fall within or close to these regions, further suggesting a tendency toward selective reporting of significant results. Such asymmetry may also be partially attributed to genuine heterogeneity stemming from differences in AI model types, datasets, or evaluation methodologies rather than publication bias alone.

The scarcity of studies in the top-left or far-left portions of the plot representing strong negative or non-significant results raises concern about potential underreporting. This has important implications, as unaddressed publication bias can lead to inflated pooled effect size estimates and overly optimistic conclusions. While contour-enhanced funnel plots improve interpretability over traditional plots by incorporating significance thresholds they remain primarily qualitative tools. In this study, Egger’s regression test confirmed statistically significant asymmetry (p < 0.001) providing quantitative evidence of potential publication bias in the included literature.

Meta-regression plot showing the publication years

The meta-regression plot generated using R version 4.4.3 provides valuable insights into how the effect sizes of studies evaluating AI-driven IDS in IoT environments have evolved over time as shown in Fig. 5. This plot, which regresses effect sizes on publication years, reveals a discernible upward trend, suggesting that more recent studies report stronger performance advantages for AI-based IDS solutions compared to traditional techniques. The red regression line indicates a positive linear relationship, meaning that as the years progress from 2022 to 2025, the reported effect sizes increase. This trend is visually supported by the spread of data points, with earlier studies generally showing lower or even negative effect sizes, while later studies tend to cluster above the zero baseline, indicating better outcomes in favor of AI. The gray shaded region surrounding the regression line represents the 95% confidence interval, which quantifies the uncertainty around the predicted effect sizes. While the interval is relatively narrow for mid-range years like 2023, it widens towards 2025, which may be attributed to a smaller number of studies in the more recent publication years or increased variability in effect size reporting. This widening band suggests that although the general trend is upward, there is some variability in how recent studies assess and report AI effectiveness, possibly due to differences in methodology, datasets, evaluation metrics, or model complexity.

The positive association between effect size and publication year may reflect genuine progress in AI technologies. In the context of IoT security, newer AI models such as deep neural networks, convolutional and recurrent architectures, and hybrid ensemble methods have been designed to tackle increasingly complex and high-dimensional data characteristic of IoT traffic. Advances in training algorithms, model optimization, and real-time processing capabilities have also contributed to more robust and adaptive intrusion detection systems. Consequently, newer studies are more likely to showcase these advancements, resulting in higher reported performance metrics compared to earlier work that relied on more conventional or rule-based approaches. This pattern may reflect a growing emphasis in the research community on benchmarking AI solutions against rigorous and diverse datasets. As the field matures, researchers are adopting more standardized evaluation frameworks and are likely to report not just accuracy, but also precision, recall, and F1-scores, providing a more holistic view of model performance. This evolution in research practices may partially explain the improved effect sizes in recent studies. The increase in computational resources and the accessibility of open-source AI libraries have lowered the barrier for deploying state-of-the-art techniques, allowing more comprehensive experimentation and evaluation in academic studies.

Artificial intelligence techniques used

The findings derived from this meta-analysis based on AI techniques applied in IDSs for IoT ecosystems predominantly fall under two overarching categories: machine learning and deep learning. Within traditional machine learning, methods such as SVM, RF, decision trees (DT), k-nearest neighbors (k-NN), naïve Bayes (NB), and gradient boosting are frequently employed (Allafi & Alzahrani, 2024; Alzubi et al., 2025; Assiri & Ragab, 2023; Ben Atitallah et al., 2024; Benaddi et al., 2022; Friha et al., 2023; Haider et al., 2024; Hamouda et al., 2024; Indra et al., 2024; Kantharaju et al., 2024; Lella et al., 2025; Mousavi, Sadeghi & Sirjani, 2023; Sadhwani et al., 2025; Saurabh et al., 2024; Sejaphala, Malele & Lugayizi, 2024; Shtayat et al., 2023; Termos et al., 2023). On the other hand, deep learning approaches commonly involve CNNs, RNNs, LSTM networks, and Autoencoders (Oseni et al., 2023). These approaches are favored across a large number of reviewed studies for their capacity to process extensive, high-dimensional data and to adapt dynamically to emerging cyberattack patterns (Allafi & Alzahrani, 2024; Assiri & Ragab, 2023; Chandnani et al., 2025; Gueye et al., 2023; Imtiaz et al., 2025; Keshk et al., 2023; Shtayat et al., 2023; Siddharthan, Deepa & Chandhar, 2022; Sneha & Prasad, 2024; Termos et al., 2024).

A notable trend in recent literature is the preference for hybrid methodologies that integrate multiple AI algorithms. This includes ensemble approaches that combine several classifiers to mitigate bias and variance, as well as layered system architectures where each stage utilizes a distinct model to enhance detection accuracy (Bar, Prasad & Sayeed, 2024; Mallidi & Ramisetty, 2025; Mishra & Pandya, 2021; Salem et al., 2024; Serrano, 2025; Siganos et al., 2023). For example, one layer might utilize random forests for initial anomaly detection, followed by deep learning models for advanced classification. These integrated systems often demonstrate improved performance across diverse evaluation metrics, as they capitalize on the unique strengths of each algorithm (Bar, Prasad & Sayeed, 2024; Manivannan, 2023; Shtayat et al., 2023; Tawfik, 2024). There has been a modest but growing interest in explainable AI (XAI) techniques within the IDS (Attique et al., 2024; Keshk et al., 2023; Lella et al., 2025; Sadhwani et al., 2025). Although most research continues to focus primarily on optimizing quantitative metrics like accuracy and precision, a subset of studies has begun to explore model interpretability to enhance transparency and trustworthiness. This movement reflects an emerging recognition that model decisions must be explainable to cybersecurity professionals and system stakeholders, especially in critical infrastructure settings (Lella et al., 2025; Rehman et al., 2025; Saheed, Omole & Sabit, 2025; Saleh et al., 2025).

Meta-analysis results by metric

This meta-analysis consolidates findings from selected studies to assess the comparative effectiveness of AI-driven IDS vs traditional methods across various IoT application domains. Traditional IDS typically depend on rule-based or signature-based mechanisms, relying on fixed attack patterns and expert-defined rules for anomaly detection. In contrast, AI-based IDS utilize machine learning and deep learning models that support adaptive data-driven detection of new and evolving threats. The aggregated evidence demonstrates a clear and statistically significant advantage in performance for AI-based approaches. Both fixed and random effects models yielded a mean accuracy difference (MD) of 0.2115, suggesting that AI-driven systems outperformed traditional IDS by an average of 21.15% in accuracy (Ahmed et al., 2025; Behera et al., 2024; Prasad et al., 2025; Tyagi et al., 2024). This performance gain was statistically robust (p < 0.0001), with 95% confidence intervals of [0.1738, 0.2492] under the fixed-effects model and [0.1176, 0.3054] under the random-effects model, corroborating earlier research supporting the superiority of AI-based IDS (Konda, Ayyannan & Chandramouli, 2023; Kulrujiphat & Kulrujiphat, 2024; Lella et al., 2025; Rasheed & Alnabhan, 2024). Substantial heterogeneity was observed across studies (I² = 82.3%; Q = 146.53, df = 26, p < 0.0001; τ² = 0.0464), likely due to variations in datasets, algorithmic models, and evaluation protocols.

However, relying on accuracy alone can be misleading particularly with imbalanced datasets common in IDS benchmarks such as NSL-KDD, CICIDS2017, ToN-IoT, BoT-IoT, and Edge-IIoTset. High accuracy may mask poor performance on minority (malicious) classes. Precision, which measures the proportion of true positives among predicted positives reflects a model’s ability to avoid false alarms metrics where algorithms like SVM and RF tend to perform well (Abdullahi et al., 2022; Lella et al., 2025; Rehman et al., 2025). Recall indicating how effectively a model identifies actual threats is especially important in high-risk environments and is typically higher in deep learning models such as RNNs and LSTMs (Bar, Prasad & Sayeed, 2024; Mousavi, Sadeghi & Sirjani, 2023; Rehman et al., 2025; Saheed, Omole & Sabit, 2025). The F1-score balances precision and recall offering a comprehensive view of detection performance. Hybrid and ensemble models often excel in this metric by better managing trade-offs between sensitivity and specificity. Other performance indicators including AUC-ROC, Matthews correlation coefficient (MCC), and detection latency are also employed to account for class imbalance and real-time requirements in IoT scenarios (Imtiaz et al., 2025; Mishra & Pandya, 2021; Sneha & Prasad, 2024).

Theoretical contributions

This meta-analytical review makes several important theoretical contributions to the growing field of AI in IDS for IoT environments. First and foremost, the study consolidates fragmented empirical findings and offers a structured overview of how different AI techniques spanning machine learning, deep learning, hybrid models, and ensemble methods perform across key evaluation metrics. This synthesis not only highlights performance trends but also contributes to theory-building by identifying patterns and relationships that are not easily observable through individual studies. By examining metrics like accuracy, precision, recall, and F1-score in tandem, the review strengthens the theoretical understanding of trade-offs between detection capabilities and error rates, which are critical when designing IDS models. Second, the study extends the theoretical discourse on model selection and optimization in cybersecurity applications. It suggests that no single AI model universally excels across all performance indicators. Rather, context-specific choices such as whether to prioritize minimizing false positives (precision) or maximizing threat detection (recall) must be informed by the specific demands and vulnerabilities of the target system. This nuanced insight adds depth to existing models of AI application in security, encouraging a more layered and adaptive approach. Third, the inclusion of meta-regression analysis to assess changes in AI-IDS performance over time adds a temporal dimension to theoretical discussions. It reflects how improvements in computational capacity, algorithmic development, and dataset quality influence outcomes thus integrating technological evolution into the theoretical framework of cybersecurity research.

Practical implications

The findings of this meta-analysis hold meaningful practical implications for cybersecurity professionals, IoT system designers, policymakers, and organizations deploying AI-based IDSs. Firstly, the consistent outperformance of AI-IDSs across key metrics especially accuracy, precision, recall, and F1-score provide strong evidence for practitioners to consider transitioning from traditional IDS solutions to AI-driven ones. These results suggest that AI models are better equipped to handle the complexity and volume of threats in modern IoT environments, which often involve vast numbers of connected devices generating high-frequency, heterogeneous data. Secondly, the variation in performance across different AI techniques offers practical guidance for model selection. For instance, organizations that prioritize minimizing false alarms may lean toward machine learning models like random forest or support vector machines, which showed higher precision. On the other hand, sectors with a low tolerance for missed threats such as healthcare or critical infrastructure may benefit more from deep learning models like RNNs or CNNs, which offer higher recall. The study also underscores the advantages of using hybrid or ensemble methods to strike a balance between these competing demands, making them ideal for complex or high-stakes environments. Third, the upward trend in performance over time highlighted by the meta-regression analysis reflects real-world advancements in AI capability and data accessibility. This evolution provides reassurance to stakeholders that AI-IDSs are not static technologies but are continually improving. However, it also signals the need for ongoing investment in skills development, infrastructure upgrades, and periodic reassessment of deployed models to ensure they remain effective. The detection of publication bias and performance inflation in smaller studies serves as a cautionary note. It encourages organizations to critically evaluate vendor claims or academic benchmarks and to conduct in-house testing where possible before full-scale deployment. Finally, the study advocates for the standardization of evaluation protocols and reporting practices, which would significantly improve practical decision-making. Clearer benchmarks would allow practitioners to compare tools more easily, assess cost-benefit trade-offs, and justify investments to stakeholders. There is interest in XAI Of the 51 studies reviewed six (Attique et al., 2024; Keshk et al., 2023; Sadhwani et al., 2025; Shtayat et al., 2023; Siganos et al., 2023; Sneha & Prasad, 2024) (approximately 11.8%) specifically focused on XAI, indicating a growing yet still limited emphasis on model interpretability in the context of IoT intrusion detection research.

Although the meta-analysis demonstrates the superior performance of AI-driven IDS models, their implementation in real-world IoT environments faces several practical obstacles. One major issue is latency, as many IoT security scenarios demand real-time or near-real-time detection to prevent the rapid spread of attacks. Another challenge is the limited energy capacity of battery-operated or resource-constrained IoT devices, which restricts the use of computationally heavy models like those based on deep learning. Additionally, hardware limitations must be taken into account, since many edge and embedded devices lack the necessary processing power and memory to support complex AI algorithms efficiently. To overcome these challenges, strategies such as lightweight model optimization, hardware acceleration (e.g., using specialized edge AI chips), and energy-efficient system design will be essential for the practical and sustainable deployment of IDS solutions.

Limitations and future directions

While this meta-analysis offers valuable insights into the effectiveness of AI-driven IDSs in IoT environments, it is important to acknowledge several limitations that also point to promising directions for future research. One key limitation is the heterogeneity among the included studies. Differences in datasets, evaluation metrics, and implementation contexts introduced variability that complicates direct comparisons and may reduce the precision of aggregated results. To enhance comparability and replicability, future research should prioritize the use of standardized benchmarking practices and publicly available, diverse IoT datasets. Publication bias presents another concern, as indicated by funnel plot asymmetry. The tendency to publish only positive or statistically significant findings may inflate perceptions of AI-IDS efficacy. To address this, future reviews should actively incorporate grey literature such as technical reports, dissertations, and conference articles to build a more comprehensive and unbiased evidence base. A further limitation is the predominance of simulation-based studies. Most AI-IDS models were evaluated using static datasets in offline environments, which may not accurately reflect the challenges of real-world IoT systems characterized by dynamic data, limited computational resources, and unpredictable network conditions. Future studies should emphasize real-time deployment and performance validation in operational IoT settings. Interpretability remains a pressing issue. While this study focused primarily on accuracy-related performance metrics, the lack of attention to model transparency can hinder adoption in security-sensitive environments. Future research should investigate how XAI techniques influence user trust, system accountability, and decision-making efficacy in cybersecurity contexts. With the rapid evolution of cyber threats including adversarial machine learning, zero-day exploits, and domain-specific vulnerabilities there is a clear need for adaptive AI-IDS models capable of responding to novel attack vectors. Continuous learning and regular model updates will be essential to ensure lasting effectiveness. The review reveals a geographic skew toward studies conducted in India, Saudi Arabia, and China, reflecting both the global distribution of publications and the accessibility of data in these fields. The limited representation from regions such as Africa, South America, and certain parts of Europe constrains the broader applicability of the findings. To enhance global relevance, future research should emphasize cross-regional studies that capture a more diverse range of contexts.