Back to all available

Artificial Intelligence for network security: Deep learning and machine learning techniques in anomaly detection systems

artificial-intelligence
data-mining-and-machine-learning
security-and-privacy
neural-networks
internet-of-things

Abstract

Background. In modern telecommunication networks that include 5G, Internet of Things, and edge computing infrastructures, large volumes of heterogeneous traffic increase exposure to attacks such as distributed denial of service and zero-day exploits. Traditional signature-based intrusion detection systems struggle to cope with the complexity, variability, and speed of these environments. Artificial intelligence, including machine learning and deep learning, offers adaptive, data-driven anomaly detection that can better protect these large-scale networks.

Methodology. This review followed a structured PRISMA-guided process. Relevant studies were identified from major digital libraries. Articles were selected based on their focus on AI-driven anomaly detection in telecommunication networks, novelty of the proposed algorithms, and availability of empirical evaluation. The final set of papers was analyzed to compare models, datasets, performance metrics, and deployment scenarios.

Results. The review shows that several AI architectures, such as convolutional neural networks, recurrent neural networks, autoencoders, and ensemble and hybrid models, are effective for detecting known and novel anomalies in core, access, and edge network segments. In most studies, deep and hybrid approaches achieved higher detection accuracy, lower false positive rates, and better adaptability to changing traffic patterns than conventional methods. However, common limitations remain, including imbalanced and outdated datasets, poor scalability to real-time high-speed traffic, limited interpretability of complex models, and challenges in deploying AI at distributed and latency-sensitive edge locations.

Conclusions. Integrating AI into telecom network security represents a significant shift toward intelligent and automated anomaly detection. Current evidence indicates that AI-based methods can strengthen resilience and support more reliable and secure communication services. Future work should focus on security for the upcoming 6G architectures, AI-enabled edge intelligence, and federated learning based collaborative anomaly detection to build decentralized, privacy-preserving, and self-protecting communication infrastructures.

Ask to review this manuscript

Notes for potential reviewers

  • Volunteering is not a guarantee that you will be asked to review. There are many reasons: reviewers must be qualified, there should be no conflicts of interest, a minimum of two reviewers have already accepted an invitation, etc.
  • This is NOT OPEN peer review. The review is single-blind, and all recommendations are sent privately to the Academic Editor handling the manuscript. All reviews are published and reviewers can choose to sign their reviews.
  • What happens after volunteering? It may be a few days before you receive an invitation to review with further instructions. You will need to accept the invitation to then become an official referee for the manuscript. If you do not receive an invitation it is for one of many possible reasons as noted above.
  • PeerJ Computer Science does not judge submissions based on subjective measures such as novelty, impact or degree of advance. Effectively, reviewers are asked to comment on whether or not the submission is scientifically and technically sound and therefore deserves to join the scientific literature. Our Peer Review criteria can be found on the "Editorial Criteria" page - reviewers are specifically asked to comment on 3 broad areas: "Basic Reporting", "Experimental Design" and "Validity of the Findings".
  • Reviewers are expected to comment in a timely, professional, and constructive manner.
  • Until the article is published, reviewers must regard all information relating to the submission as strictly confidential.
  • When submitting a review, reviewers are given the option to "sign" their review (i.e. to associate their name with their comments). Otherwise, all review comments remain anonymous.
  • All reviews of published articles are published. This includes manuscript files, peer review comments, author rebuttals and revised materials.
  • Each time a decision is made by the Academic Editor, each reviewer will receive a copy of the Decision Letter (which will include the comments of all reviewers).

If you have any questions about submitting your review, please email us at [email protected].