An intelligent anomaly detection system for IoT using a hybrid metaheuristic evolutionary strategy

IoT

IoT connects smart settings, IP-based devices, and other devices, enabling communication without human intervention. Techniques include smart manufacturing, smart agriculture, intelligent cities, power, energy, and logistics, with an estimated 75 billion connected devices by 2025 (Lele, 2018). Applications include smart logistics, the intelligent industry, healthcare, control, surveillance, management, and smart or intelligent homes (Anthi et al., 2019).

IoT devices have been the target of numerous cybersecurity incidents, with 66% of industrial manufacturer sectors experiencing such incidents in the past 2 years (Noman, Mujahid & Fatima, 2021). Examples include the STUXNET worm attack against Iran’s nuclear enrichment facilities, smart power infrastructure in Ukraine, and distributed denial of service (DDoS) attacks on the DYN domain name system (DNS). Telecommunications providers, including device makers, IoT applications, and evolved packet core (EPC), must ensure consumer protection, security, and accessibility to services in the IoT environment.

IoT security

Before creating an IDS, it is important to gain prior basic knowledge of the IoT atmosphere’s challenges that affect its safety (Lazarevic, Kumar & Srivastava, 2005). Some of the most prominent IoT safety attributes are integrity, confidentiality, and authentication.

The article is structured as follows: “Literature Review” reviews foundational concepts and recent literature. “Proposed Hybrid Evolutionary Model” details the proposed methodology. “Results and Discussion” presents and discusses the results. Finally, “Conclusion” concludes the study.

IoT attacks

IoT consumers and manufacturers are more conscious of IoT products now than they were in the past due to various attacks on the IoT system in recent years (Ge et al., 2019). Several IoT system attacks are described in this section. IoT devices, connected to wireless networks, are vulnerable to physical and cyberattacks, posing risks to users and posing potential security threats (Mahmood et al., 2025).

A targeted attack on a web server aims to limit authentic users’ access to targeted services, reducing device availability. This attack is common in sensor-based applications with constrained resources, particularly in the IoT environment, where hackers trick the server into responding more frequently. Hole attacks, spoofing, sybil, and man-in-the-middle (MITM) attacks are methods used by attackers to manipulate data, posing a significant threat to system security (Ashraf et al., 2023). DOS attacks cause system disruptions, preventing users from accessing IoT devices, affecting decision-making, and extending battery life by constantly turning devices on (Soomro et al., 2024). MITM attacks identify faulty data in communication, which can lead to intercepts and disruptions, as original data can be easily hacked, and false information can be inserted (Ashraf, Sohail & Yousaf, 2023). Spoofing and sybil attacks: IoT attacks often target user identity via Medium Access Control (MAC) and radio frequency identification (RFID) addresses, posing a threat to the service due to the lack of robust security protocols (Abomhara & Køien, 2015).

• Hole attacks: active assaults, such as gray hole and blackhole attacks, degrade network functionality and cause network crashes (Sunitha & Latha, 2025).

• Jamming attack: IoT devices are becoming more active due to unwanted signals, which can make their performance worse due to increased memory and bandwidth usage (Elmasry, Akbulut & Zaim, 2020).

• Malicious input attack: IoT devices are vulnerable to malicious input assaults, including trojans, rootkits, worms, adware, and viruses, which can reduce wireless network output, causing financial and power loss.

• Data tampering: data tampering poses a significant risk to organizations, posing potential damage and requiring immediate attention to prevent such attacks.

In Table 1, a lot of work is still carried on to achieve the classification accuracy to detect anomalies for the IoT networks, which can further decrease the computing cost and time of prediction. A lot of the research has been done in this part for designing anomaly detection systems for the IoT networks; however, these depend on a combination of an FS optimization algorithm or on a benchmark PSO-based technique. This work provides an efficient and intelligent method that fills a gap in the literature by using fewer parameters and achieving comparable or higher accuracy while incurring lower processing costs and requiring less prediction time.

Thorough review of related works

Several studies have addressed IoT intrusion detection using the BoTNeTIoT dataset, focusing on feature selection, machine learning (ML), and deep learning (DL) techniques. Table 2 has a summary of key works, their contributions, limitations, and how our proposed approach overcomes these challenges.

Expanded review of related works (2022–2025)

This section critically analyzes recent works (2022–2025) on IoT intrusion detection, focusing on methodologies, datasets, and limitations. We categorize them in Table 3 based on machine learning (ML), deep learning (DL), hybrid models, and feature selection techniques, highlighting gaps in the work done. Existing works excel in specific attack types (e.g., DoS) but struggle with diverse IoT threats. Our approach integrates DL, optimized features, and XAI to overcome these limitations, ensuring higher accuracy, interpretability, and real-time performance. The comparative work is summarized in Table 4.

Whale optimization-based random forest algorithm

A hybrid model combines the anomaly and the signature based IDS in order to offer the best solution for the storage and for the computing expenses while lowering false positive alert rates. The majority of systems currently employ hybrid IDS due to its effective detection and simpler operation.

In this section, the flowchart and algorithms of WOA-based random forest are discussed. with a set of random answers as a base case, the WOA algorithm begins. the positions of the search agents are updated after each iteration about either the best result so far or a randomly selected search agent. Exploration and exploitation are provided by decreasing the value from 2 to 0. When $|\overrightarrow{A}|<1$ for updating the search agents’ positions, the best option is chosen, and when $|\overrightarrow{A}|>1$, then choose a random search agent. The WOA algorithm have a termination criteria. The WOA algorithm’s pseudocode is described below. The pseudocode is explained with each equation.

To address intrusion detection in IoT using the WOA, the proposed hybrid approach optimizes feature selection and classifier hyperparameters. Here is the structured algorithm:

The algorithm successfully accelerates the rate of optimization, The WOA’s search methodology also has some advantages in some problems. The data preprocessing and problem formulation steps usually consist of three to four steps. Data imputation step can be added in any stage.

Problem formulation

WOA steps

After the candidate population initialization, objective or fitness function is applied, like in a typical genetic algorithm. The WOA goes through encircling the prey, Spiraling the bubble and searching for the prey.

Computational complexity

The computational complexity of the WOA is primarily governed by three parameters: the population size N, the number of iterations T, and the dimensionality of the search space D. The overall time complexity per run can be expressed as:

$$\mathcal{O}(T\cdot N\cdot D).$$

This reflects the cost of fitness evaluation and position updates for all individuals in the population across all iterations. In terms of space requirements, WOA maintains a population of N candidate solutions, each of dimensionality D, resulting in a space complexity of:

$$\mathcal{O}(N\cdot D).$$

This storage is needed for retaining particle positions and auxiliary coefficient vectors (e.g., $\mathbf{A}$, $\mathbf{C}$) used in the update equations.

The mathematical formulation of the proposed algorithm employs conventional symbolic notation, with all variables and operators defined explicitly in relation to their corresponding equations. Each equation is accompanied by descriptive explanations to promote clear understanding of the algorithm’s internal dynamics and the mathematical relationships they represent.

Random forest

For categorization, Random Forest (RF) is employed. an ensemble technique called RF employs a number of classifiers with tree structures. Each tree is constructed using a decision tree technique, using a different bootstrap sample from the original data, and only selecting a small number of features for the split at each node of the tree. An objective estimate of generalization error called Out-Of-Bag (OOB) assessment is carried out on the learning samples that weren’t selected by bootstrapping. After the forest has been constructed, a new sample is given into each tree for classification. Each tree then casts a unit vote for a certain class, indicating its opinion. When compared to traditional ml classifiers, ensemble classifiers are one way to build a powerful classifier with increased classification accuracy. the model’s mathematical expression can be seen in Eq. (8).

(8) $$C(X)=\mathrm{S}\mathrm{i}\mathrm{g}\mathrm{n}\left(\sum _{J=1}^M{C}_J(X)\right),$$where M is the total number of classifiers participating in the classification or vote and J is the number of classifiers participating individually. The benefits of RF in organized data include its superb accurate performance. It can operate on huge datasets with many dimensions and is computationally efficient. Most of the time, it does not overfit, and it is also noise-resistant. Unbalanced datasets can be handled by it.

Experimental setup and matrices for evaluation

A computer running Microsoft Windows 10 or 11 with at least an Intel(R) Core(TM) i7-6500U processor with two cores, four logical processors, and 16 GB of RAM was used to test the performance of the proposed model. Python is used to create feature selection and classification algorithms (version 3.8). Anaconda Navigator is set up on the aforementioned system for the experimental setting. Accuracy, precision, recall, and F1-score matrices were used to evaluate the performance of hybrid evolutionary strategy. WOA is like other metaheuristic evolutionary algorithms. It is always used in hybrid format with ordinary lightweight AI classifiers. It takes into account all candidate solutions, so computationally a bit expensive but ensures perfect cyber security. Confusion matrices provide a visual representation of performance in distinguishing between normal and intruder or malicious node. These evaluation metrics helped in understanding the strengths and weaknesses of each model for the detection of intruder or identification of malicious IoT nodes.

Accuracy: Accuracy measures how closely experimental results match predefined true values. It is calculated as:

(9) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{TP+TN}{TP+FP+FN+TN},$$where: TP = True Positives, TN = True Negatives, FP = False Positives, FN = False Negatives.

Simulation results

The simulation results of the suggested technique utilizing the BoTNeTIoT dataset were briefly presented in this section. This work selected the BoTNeTIoT dataset for our model’s training and testing because it is one of the most recent datasets to be gathered in the IoT environment. The BoTNeTIoT dataset was created in 2020. There are 83 network characteristics, 625,783 records and three label features in the entire dataset. Five distinct classes—mirai, scan, DoS, normal, and MITM—are used to group the total number of records. Once more, these classes are further divided into seven subclasses: mirai brute force, mirai HTTP flooding, mirai UDP flooding, scan host port, scan port OS, SYN flooding, and ARP spoofing. These traits are as shown in Table 5. Three different types of label characteristics are present in this dataset: binary, category, and sub-category.

Scan, mirai, DoS assaults, and MITM attacks are the four primary types of attacks. Tables 6 and 7 list these attacks along with their subdivisions.

Performance evaluation

This section describes how the suggested model’s performance was evaluated. The BoTNeTIoT dataset was employed in the experiment, and the train-test split validation technique was used to thoroughly evaluate the ML algorithms’ performance. BoTNeTIoT has 625,783 occurrences. 30% of the data was utilized to verify the model, while 70% was used for training.

Binary classification performance of BoTNeTIoT dataset

The confusion matrix and convergence matrix for the performance of binary classification on BoTNeTIoT are shown in Figs. 3 and 4.

Category classification performance of BoTNeTIoT dataset

The category classification performance for BoTNeTIoT dataset based on the parameters i.e., Pr, Rc and F1-score are shown in Table 7. The confusion matrix and convergence matrix for the performance of category classification on BoTNeTIoT are shown in Fig. 5. The accuracy for the performance of binary classification and category classification on BoTNeTIoT is shown in Fig. 6.

Table 8 shows the performance of four models (LR, SVM, WOA, and ANN) in cases of different kinds of attacks. In some IDS datasets, the types of attacks are not clear. WOA performance is comparatively better than the other three.

Table 8:

Detection accuracy of models across attack Types.

Types of attack	LR (%)	SVM (%)	WOA (%)	ANN (%)
DoS-S.F	100	100	100	100
M.ARP.S	93	92	99	98
M.A.F	87	87	88	90
M.HTTP.F	87	87	93	90
M. Host B force	87	87	100	98
M. UDP. F	95	95	98	96
Normal	97	97	100	100
SHP	86	88	96	95
S.P OS	90	91	98	97

DOI: 10.7717/peerj-cs.3334/table-8

Figure 7 demonstrates accuracy for each of the four tests carried out using the BoTNeTIoT dataset. These findings demonstrate the viability and effectiveness of our method for identifying malicious and benign (normal) nodes utilizing the four algorithms.

Figure 8 shows and discusses the ROC curves of (1) LR, (2) SVM, (3) WOA, and (4) ANN. For the three attack types—mirai-ack flooding (M.A.F), mirai-http flooding (M.Http.F), and mirai-udp flooding (M.Udp.F)—the artificial neural network (ANN) model yielded the best detection rate (Dr). However, for MITM ARP spoofing (M.Arp.S), mirai-host brute force (M.Host.B.Force), scan host-port (S.H.P), and scan port OS (S.P.OS), the WOA model yielded the highest detection rate (Dr).

Discussion of overfitting/high scores in Table 8

The near-perfect scores (e.g., 100% for DoS-S.F across all models) in Table 8 may suggest overfitting or data leakage, especially if the test set was not strictly isolated during preprocessing. While high performance on attacks like DoS-S.F could reflect separable features, the consistency of 100% scores warrants scrutiny—particularly for simpler models (logistic regression (LR), SVM) that may lack the complexity to generalize perfectly. For minority classes (e.g., Scan Host Port at 86–96%), the lower but still elevated metrics could indicate imbalance-driven bias, where oversampling (e.g., SMOTE) artificially boosts scores without proportional real-world gains. To validate robustness, future work should include: (1) cross-validation with independent test sets, (2) confusion matrices to check for minority-class overestimation, and (3) synthetic noise injection to test model resilience.