Software defined network intrusion system to detect malicious attacks in computer Internet of Things security using deep extractor supervised random forest technique

Challenges

Many areas of research have concentrated on attack detection (Zaman et al., 2025), reliability, privacy and data security (Wani, S & Khaliq, 2021). Nevertheless, many classical models insufficiently represent critical sequential patterns, while many deep learning methods necessitate substantial network training, resulting in costly delays in attack detection and reduced affordability in real-time applications (Adamou Djergou, Maleh & Mounir, 2021). Consequently, it is important to employ advanced methodologies to investigate these issues. In a vast, dynamic environment, creating a security system that reliably discerns between normal and abnormal behavior is a challenging undertaking (Deb & Roy, 2022). The challenge in identifying features that may differentiate normal from DDoS attacks arises from the multitude of diverse datasets and different methods (Khamaiseh et al., 2022). The limited size of the training set results in a lack of low-frequency attacks (Wang et al., 2024). This limitation affects the evaluation of the proposed IDS using a real-world dataset that simulates DDoS, web attacks, and BotNet attacks. The high computational and resource requirements of many deep learning and conventional prediction models make network congestion in SDN settings more challenging. The performance of the real-time network may be degraded, particularly when there is heavy traffic.

Problem statement and contributions

Traditional approaches in some cases unable to effectively detect temporal patterns in network traffic and experience significant overhead, particularly in dynamic settings such as SDN, leading to inadequate performance, especially against malicious or DoS attacks. Systems are unreliable and perform poorly due to the difficulty of tracking massive volumes of unclassified data. Despite its limitations, machine learning may be useful when integrated with NIDS (network intrusion detection system) to detect and classify threats. Our proposed model increases the accuracy of detection for different attack modes by using traffic trend analysis over time. It minimizes the number of false positives by using ongoing behavioral training to spot common and unusual patterns. Real-time attack identification is made easier by integrating the proposed model into SDN environment. Deep learning is a novel approach to estimating the likelihood of finding solutions to machine learning problems. The main contributions are as follows:

• The existing model has issues with complicated architecture, large batch sizes, high training processing, and slower SDN networks that lead to congestion. In contrast, our model is lightweight, uses fewer GPU resources, and provides a more accurate and adaptable detection method without causing excessive network congestion.

• The proposed methodology extracts temporal, long- and short-term flow behaviors with dynamic, complex network interactions, and the features are then fed into a supervised random forest model for detecting and mitigating DDoS attacks.

• To address a severe class imbalance across attack types, we employed Synthetic Minority Oversampling Technique (SMOTE), which significantly reduced misclassification errors and improved generalizations. Experimental results show that after using SMOTE, the classification errors went down from 5 to 0 for Brute Force Attack (BFA) attacks and from 6 to 0 for web attacks, proving that balancing the training data in SDN really works.

• We validate our proposed model by adding another intrusion dataset that has different types of attacks, showing that the model can work well in various intrusion situations. Furthermore, we employed an explainable AI technique to explain model decisions, improving reliability and transparency in security-critical environments.

• The proposed model enhances the detection rate of infrequent attacks and demonstrates superior performance compared to state-of-the-art models while minimizing false positive or negative rates. The proposed method for efficient SDN attack detection and classification is thoroughly discussed and evaluated.

Dataset details

InSDN is an extensive dataset specifically designed for assessing intrusion detection systems in the context of SDN. The recently published dataset comprises a diverse array of assaults, including both harmless and harmful ones, that have the potential to target several standard components of SDN. InSDN considers a range of threats, including DoS, DDoS, brute force attacks, web application vulnerabilities, exploitation attempts, probes, and botnets. The created data is often sent over many widely used application services, such as email, File Transfer Protocol (FTP), Secure Shell (SSH), Domain Name System (DSH), Hypertext Transfer Protocol Secure (HTTPS), Hypertext Transfer Protocol (HTTP), Secure Sockets Layer (SSL), and others. We generated the dataset using four virtual machines. The first virtual computer serves as a substitute for the malevolent server running Kali Linux. The ONOS controller connects to the secondary system, an Ubuntu 16.4 server. The third server is an Ubuntu 16.4 system that is compatible with both OVS and Mininet switches. In addition, we used the same OVS technology to create the Damn Vulnerable Web Application Server (DVWA) inside a Docker container. We also evaluated many potential attack scenarios that may arise from both inside and outside the SDN network. The OVS_Group is a fixed representation of the OVS server’s assault recordings. The six components of the course address various topics like web assaults, brute force attacks, botnets, distributed denial of service, and probing attacks (Badcodebuilder, 2023). The dataset records are shown in Fig. 2.

Data preprocessing

Data preparation is the process of modifying the values in a dataset in order to improve data collection and manipulation. Because there is such a large discrepancy between the dataset’s top and lowest values, normalizing the data makes the method go more smoothly. According to reference (Yadav & Kalpana, 2019), there are substantial advantages to data normalization for neural network algorithm classification. Training neural networks utilizing the back-propagation technique becomes much faster with input value normalization applied, resulting in a more efficient neural network. Dataset processing begins with removing null values and empty cells. To prevent any potential damage to the model, we eliminated rows with empty data. When training on massive datasets, it may not always be a good idea to use every feature. Some features of the dataset can influence the outcome negatively. By obtaining the dataset’s correlation matrix, we may identify superfluous features and eliminate those with high correlation values (Yang et al., 2022).

Reshape data

Data reshaping modifies the configuration of data from one format to another to enable analysis or subsequent processing. During this phase, rows and columns may be reorganized, the data type may be modified, or the format may shift from wide to long. The numpy.reshape() function can change the shape of a numpy array without modifying its data. To maintain compatibility of arrays with supplementary operations, it is often necessary to alter their geometry. The np.reshape() function modifies the array’s shape and thereafter returns the array in its new configuration. The only requirement is that both the original array and the updated array possess the same number of elements.

Normalization

Data designed for machine or deep learning is frequently normalized before applications. Normalization is the procedure of ensuring uniformity in the dimensions of all columns within a dataset. Normalizing each sample is not requisite for machine learning. It is crucial when the feature ranges vary. The use of normalized data enhances both the accuracy and utility of a model. A prevalent method of discussing min-max scaling is to achieve uniformity. Modification of the features occurs within a specific range, typically between 0 and 1. This is advantageous in scenarios where upholding the original range is essential and remains inside the specified range, allowing for the original figures to be comprehended.

Label encoder

A prominent technique in artificial intelligence is to transform category data into a numerical representation that systems can readily supervise. An effective method for doing this is by label encoding, where each category item is assigned an integer value based on its alphabetical order. This strategy is direct and entails transforming every value in a column into a numerical representation (Jia & Zhang, 2021). The description of how to performed label encoder, reshape of data and normalization are shown in Snipped Fig. 3.

Data splitting

To split the data into a training set and a testing set, the study used the (train_test_split)() function as illustrated in Fig. 4. It is necessary to categorize data into two distinct groups: attributes (X) and labels (Y). The dataset has four variables: $X_{t}rain$, $X_{t}est$, $y_{t}rain$, and $y_{t}est$(). Using the $X_{t}rain$ and $y_{t}rain$ datasets, perform model training and fine-tuning. Evaluate the model’s output and label predictions by comparing them with the $X_{t}est$ and $y_{t}est$ sets. We have the option to conduct direct tests on the percentages of the training or test set. It is often advisable to ensure that training sets are larger than our test sets. A specific collection of data known as the training dataset shapes the model. The model receives instructions from the dataset. Based on its observations, the model learns new knowledge (Palimote, Atu & Osuigbo, 2021). The import statements for the Pandas, Scikit-Learn, and Numpy libraries are imported. X encompasses the characteristics, whereas Y encompasses the labels. After dividing the dataframe into two equal halves, we performed a train-test split on both the X and Y halves. In order to ensure the consistency of data, the $rando{m}_{s}tate$ parameter functions as a seed for numpy.

Proposed hybrid LSTM-SRF architecture

An LSTM utilizes ways to handle both short-term memory and long-term memory. LSTMs use gates to enhance and expedite the computational process. Recurrent neural networks (RNNs) trained using backpropagation are notorious for their struggles in learning long-term dependencies due to the issue of vanishing and exploding gradients. To mitigate the visibility of expanding gradients, one may use gradient cropping. However, addressing the issue of fading gradients requires a more intricate approach. The LSTM model proposed by Hochreiter & Schmidhuber (1997) was an early and successful solution to the problem of vanishing gradients. While LSTMs use memory cells instead of ordinary recurrent nodes, their behavior is largely similar to that of conventional recurrent neural networks. Each node in the network corresponds to the state of an internal memory cell. There is a single recurrent edge with a constant weight between this node and itself. Because of this design, the gradient may require many time steps to propagate in order to avoid becoming too small or large. Figure 5 presents the proposed deep extractor framework based on deep and supervised RF (random forest) technique for best performance in attack detection.

This concept gives rise to a “long, short-term memory”. Weights can help simple recurrent neural networks preserve their memories over time. Training gradually adjusts the weights to collect and display a diverse range of data-related information. Transient activation, which transfers across nodes, contributes to their short-term memory. A memory cell serves as an additional storage component inside the LSTM architecture. The intricate configuration of a memory cell is determined by the arrangement of basic nodes interconnected in a certain way with multiplicative nodes. LSTMs specifically aim to reduce long-term dependence. The capacity to retain information over an extended period is mostly an inherent inclination rather than a skill that needs deliberate cultivation. The sequence of modules that make up a recurrent neural network share the same characteristics as any other neural network. The primary element of traditional RNNs is a basic repeating module consisting of a single tanh layer.

Models training

We constructed the proposed model and tested its intrusion detection performance using an Anaconda Jupyter Notebook on an Intel Core-i7 CPU with 16 GB of RAM and the Windows 10 environment. When we set the shuffle parameter to True, the shuffling process discloses our information. To begin the process, we use the $X_{t}rain$ and $X_{t}est$ datasets to train and optimize our model. Subsequently, we subjected it to rigorous testing to evaluate its performance. We used a 64-unit batch size, a 0.001 kernel regularizer, dropouts, and 100 epochs. Details about hyperparameters, activation functions, and optimizers for deep learning models is illustrated in Table 1.

Evaluation

We evaluated the performance using several metrics such as the confusion matrix, ROC-AUC (Sainis, Srivastava & Singh, 2018), accuracy, precision, recall, F1-score, and precision-recall curves. We conducted an analysis and comparison of the proposed system’s performance with that of other systems. The classifier only relies on the frequency of correct predictions as the sole statistic it evaluates. One way to measure accuracy is to calculate the ratio of correct predictions to the total number of forecasts made. A precision is one that yields the anticipated ratio of correct positive outcomes to the total number of actual positive outcomes. A label’s recall is defined as the ratio of the total number of genuine positives to the number of true positives. The F1-score applies a more severe penalty to outliers. When the costs of false negatives (FN) and false positives (FP) are equal, the F1-score may serve as a reliable indicator for assessment. The true negative rate is noticeably higher.

(1) $$Accuracy=\frac{IDTP+IDTN}{IDTP+IDFN+IDFP+IDTN}$$

(2) $$Precision\text{-}PR=\frac{IDTP}{IDTP+IDFP}$$

(3) $$Recall\text{-}RE=\frac{IDTP}{IDTP+IDFN}$$

(4) $$\text{F1-Score}=2\times \frac{PR\times RE}{PR+RE}.$$

Classification performance using RNN

To get the macro average of a particular metric, it is necessary to calculate the metric for all labels and then calculate the average without considering the percentage of each label in the dataset. To compute the average for a given set of labels, one must determine the metric on all label in the collection and then aggregate the number and percentage of all the label to get the weighted average of that metric. The performance of the deep RNN approach on the SDN dataset is shown in Table 2. All performance metrics are assessed in the experiments to evaluate the effectiveness and accuracy of the models in detecting network attacks. The BFA label achieved very poor scores in all categories. The BoTNET assaults achieved a performance of 100% utilizing accuracy, recall, and F1-score. Additionally, it made 29 positive predictions out of 29, with no erroneous predictions. The RNN approach attained an accuracy of 99.4% and a macro average of 97.1%.

Classification performance using GRU

Table 3 gives an overview of how well the deep GRU technique performed on the SDN dataset. During the experiments, each and every performance measure is evaluated in order to determine how successful and accurate the models are in identifying network threats. In each and every category, the Web-Attack label received very poorly received evaluations. When it comes to accuracy, recall, and F1-score, the BoTNET attacks were able to attain a good performance. Additionally, it produced 29 accurate forecasts out of a total of 29, and it did not make any incorrect predictions. Using the GRU method, we were able to get an accuracy of 99.8% and a macro average of 95.2%.

Classification performance using BiLSTM

Table 4 describes the results of the deep BiLSTM method on the SDN dataset. For the purpose of determining the models’ efficacy and precision in detecting network threats, the tests assess all performance metrics. Reviews for the BFA-Attack label were mostly negative across the board. The accuracy, recall, and F1-score were all positively affected by the BoTNET assaults. We achieved a macro average accuracy of 94.1% and a precision of 99.5% by using the BiLSTM technique.

Classification performance using proposed LSTM

Table 5 presents the results of the deep LSTM method applied to the SDN dataset. The DDoS attack obtained a precision rate of 99.98%, the BFA achieved a F1-score of 93.91%, the DoS achieved a F1-score of 99.93%, and the Web-assault earned a recall score of 54.29%. The LSTM model attained a macro recall of 93.26% and a F1-score of 94.34%. Additionally, method achieved weighted average performance of 99.90%.

Classification performance using the proposed model

After the deep experiments, we see that LSTM method perfroms best, so we extract features from the leightweight LSTM network, and design supervised random forest model for the intrusion detection. Random Forest generates several bootstrap samples from the core dataset. Randomly swapping data points creates samples. The multiple subsets created by this approach may cause data differences in different trees. Decision trees do predictions by building on each other. Each tree puts in a “vote” for a class, and the class with the most votes obtains. Since each decision tree learns from a different data source, random forests may combine their predictions. Combination in prediction and voting in classification decrease tree variance in Random Forest, improving prediction accuracy. The results of the proposed method applied to the SDN dataset are shown in Table 6. The DDoS attack had an accuracy rate of 100%, the BFA had a recall score of 97.93%, the DoS had a precision score of 99.98%, and the Web-assault had a recall score of 82.86%. The model achieved a macro recall of 97.23% and a F1-score of 96.55%. In addition, the approach attained a weighted average performance of 99.93%.

There are four different multiclass classification methods: RNN, GRU, BiLSTM, and LSTM. Figure 6 depicts the training accuracy and loss graphs for each of these techniques. The results shown in Fig. 6 show that the LSTM outperformed the RNN in terms of the accuracy of its training. An increase in the number of epochs led to an improvement in the accuracy of the models. The figures also feature loss graphs, which are included in order to provide a better understanding of the efficacy of the model.

Figure 7 displays the precision and recall curves for the multiclass classification of the proposed model architecture. The proposed approach successfully classified network connection records as either malicious or legitimate. The suggested strategy has shown to be successful for the majority of attack types in the multiclass category. This demonstrates the credibility of the proposed method and its ability to enhance the accuracy of multiclass classification.

TSNE visualization

Deep learning models can consist of several hidden layers, which might be a challenge when it comes to understanding them. These models may be likened to a “black box,” concealing the internal mechanisms by which they analyze incoming data in order to provide the most effective outcomes. In order to get a more comprehensive understanding of how deep learning models acquire knowledge from various hidden layers, a technique called t-distributed stochastic neighbor embedding (t-SNE) may be used for feature visualization (Hamid & Sugumaran, 2020). The t-SNE visualization method, similar to principal component analysis (PCA), reduces high-dimensional data to lower-dimensional values and displays them in two dimensions (Chapagain et al., 2022). We used features extracted from the penultimate layer of the proposed model for both binary and multiclass classification. Figures 8 and 9 depict the graphs for multiclass classification. These graphs illustrate the two dimensions of the inputs used in t-SNE. The clusters in which the assault and typical traffic data occurred exhibited little overlap, as seen in Fig. 8. The model’s ability to differentiate between malicious and benign traffic is clearly shown here. Figure 10 illustrates the classification of various assaults based on their distinct characteristics. Conversely, the model may have learned several attack-specific feature patterns, since most forms of assault divide the clusters into two or more.

Confusion matrix

Confusion matrix are numerical arrays that highlight model performance. Classification models predict performance in each category (Sokkalingam & Ramakrishnan, 2022). Confusion matrix link results to initial data categories. This data shows that confusion matrix are beneficial in supervised learning when the output distribution is known. The confusion matrix simplifies classifier accuracy estimates for broad and specialized categories. It also helps identify additional essential factors that model developers use to evaluate the work. A confusion matrix may enable learning, which combines numerous models on the test set of the same dataset to provide the best results as seen in Fig. 10. Even with several classes, the confusion matrix ideas stay the same.

Experiments with additional dataset

We further evaluate the efficacy of the proposed model using additional multi-class datasets, each comprising five classes. Table 7 demonstrate that web-based attacks such as XSS and SQL injection earned a stunning precision score of 100%, while DDoS attacks recorded 99.98% and benign instances achieved 99.87%. Conversely, brute force attacks yielded a precision of only 46.09%. The DDoS attack achieved a 99.94% F1-score and a 95% weighted F1-score. The model demonstrated superior performance on an alternative multi-class dataset.

Proposed model results using SMOTE technique

The results obtained from the balanced InSDN dataset using SMOTE technique is illustrated in Table 8. SMOTE is an abbreviation for the synthetic minority over-sampling technique. This methodology is an oversampling strategy that produces synthetic minority class samples to balance the dataset. SMOTE is a technique for augmenting data by employing interpolation between instances of minority classes to create new synthetic samples.

Synthetic samples are produced by SMOTE to create links among proximate neighbors in the feature space. SMOTE produces new minority class samples by repeatedly transitioning from a minority class sample to one of its k nearest neighbors in the feature space, with k being a parameter of the algorithm.

Comparison with existing state of the art work

Table 9 presents a comparison with our research results and those of other scholars who have examined intrusion detection systems in SDN networks. ML/DL models and risk classification in SDN networks need the use of numerous datasets. Ravi, Chaganti & Alazab (2022) utilised the deep learning GRU approach for attack classification with the SDN dataset, attaining an accuracy of 98%. Said, Sabir & Askerzade (2023) employed a hybrid approach combining CNN and BiLSTM in an SDN environment, achieving an accuracy of 97%. Arthi, Krishnaveni & Zeadally (2024) also employed a hybrid model, specifically DNN-SVM, in an SDN context and attained 96% accuracy. Tang et al. (2020) utilised the NSL-KDD dataset for intrusion detection system (IDS) attack detection and attained a notably poor accuracy. Chaganti et al. (2023) used simple LSTM model for the classification of SDN attacks. The authors employed two datasets, one with two classes and the second with five classes, for the classification. The LSTM attained low accuracy and did not explain the preprocessing techniques well. Our proposed model developed a hybrid approach, LSTM-SRF, to extract features using deep learning and make detections using the SRF model. Also, we utilized two datasets, one with five classes and the second with seven classes, which is different from the mentioned work. Our proposed model attained superior performance and uses fewer resources, making it computationally efficient. Unlike previous studies that used SDN networks for training and testing purposes, our research achieved a remarkable accuracy rate of 99% in classifying attacks.

Explainable artificial intelligence

We employed explainable artificial intelligence (XAI) techniques such as SHAP. SHAP aims to enhance the comprehensibility and interpretability of AI models for humans. The objective of SHAP is to elucidate the mechanisms by which models make decisions, enabling professionals to comprehend the rationale underlying forecasts, classifications, and recommendations. These strategies facilitate our understanding of the prediction mechanisms employed by black-box models. Explainability of the proposed model using SHAP is shown in Fig. 11.

Limitations and future work

Although the main focus of our article is on detecting IoT network intrusion attacks using deep learning, we ensure that it is crucial to integrate security methods with the SDN control layer to ensure robust security. To address such an issue in future work, we plan to introduce distributed SDN architectures to reduce the number of single-point nodes. Furthermore, the development of blockchain technology or security verification networks will improve the security of the regulatory platform.

The adversarial attacks seriously bypass deep learning-based models. Although our main objective was not adversarial, we will address this limitation in the future. Future research should also investigate adversarial examples and robust learning techniques to increase model resilience to new assumptions meant to evade detection.

We believe that one of the challenges facing the cyberthreats is the ever-changing nature of attacks. Such developments could make traditional IDS models obsolete over time. Although our study uses publicly available InSDN and other data sources, we ensure that continuous improvements are essential for long-term success. Future research will use customised training materials and federated learning techniques to adjust the model to new and developing threats. Additionally, we see potential in federated learning that allows multiple tools to collaborate on standardisation while also ensuring that systems remain up to date and privacy is maintained in the context of unauthorised data storage.