An intrusion detection system based on convolution neural network

Convolution layer

Convolutional layers constitute indispensable components in deep learning, primarily tailored for processing images, videos, and other two-dimensional data (Yamashita et al., 2018). They serve the critical function of extracting features from input data and find widespread application across diverse domains. Central to convolutional layers is the convolution operation, a fundamental technique that executes local weighted summation on input data, thereby capturing spatial structural information.

The distinctive advantage of convolutional layers lies in their efficacy at extracting features from input data through the utilization of local connections and parameter sharing. Local connections entail filters convolving solely with local regions of the input data, as opposed to the entire input. Meanwhile, parameter sharing dictates that each filter’s parameters are shared across different locations in the input, employing the same set of weights. These strategies contribute to reducing the model’s storage requirements while bolstering its generalization capability.

Moreover, convolutional layers have the capacity to generate multiple feature maps by employing multiple filters, with each feature map corresponding to a specific filter. This multi-feature map approach enables the capture of variations across different feature directions within the input data, thereby facilitating the extraction of richer feature representations. By stacking multiple convolutional layers, the network progressively extracts higher-level abstract features, facilitating tasks such as classification, regression, and more in subsequent fully connected layers (FCLs) or output layers.

In essence, convolutional layers play an indispensable role in deep learning by efficiently extracting features from input data and exerting a pivotal influence across various domains. Through the mechanisms of local connections, parameter sharing, and multi-channel outputs, convolutional layers empower deep learning models to capture spatial structural information from input data and furnish rich feature representations.

The formula of convolution layer is as follows: (1)${\displaystyle y_{i,j,k}=\sum _{u=0}^{k-1}\sum _{v=0}^{k-1}\sum _{c=0}^{C-1}{w}_{u,v,c,k}{x}_{\left(i+u\right),\left(j+v\right),c}.}$

In this case, y_i,j,k represents the convolutional output of the k − th kernel at position $\left(i,j\right)$, w_u,v,c,k represents the weight at position $\left(u,v\right)$ of the c − thchannel of the k − th kernel, and x_{(i+u),(j+v),c} represents the value of the c − th channel at position $\left(i,j\right)$ in the input data.

Pool layer

Pooling layers are integral components of CNNs and hold significant academic value in image processing and computer vision domains (Nasr-Esfahani et al., 2019). Their primary function is to reduce the spatial dimensions of feature maps through downsampling operations, thereby diminishing the number of parameters and computational complexity. This dimensionality reduction not only enhances computational efficiency but also facilitates the extraction of primary features from the input data.

Within pooling layers, the most prevalent operations are max pooling and average pooling. Max pooling retains the most significant features of input regions by selecting the maximum value within each subregion, while average pooling calculates the average value of features within each subregion, thereby blurring details and mitigating the influence of noise. Both of these pooling operations contribute to extracting crucial information from the input data, providing more robust and abstract feature representations for subsequent layers. This article primarily adopts the max pooling method.

Furthermore, pooling layers exhibit translation invariance, meaning that irrespective of the features’ positions in the input data, the pooling operation can still detect their presence and effectively aggregate them. This characteristic renders pooling layers resilient to translation, rotation, and scale changes in images, thereby enhancing the model’s generalization ability (Mumuni & Mumuni, 2021; Zhang et al., 2019).

In summary, pooling layers play an essential role in CNNs, offering potent feature extraction and abstraction capabilities for image processing tasks by reducing spatial dimensions, extracting primary features, and providing translation invariance. In both academic research and practical applications, the design and optimization of pooling layers are crucial for enhancing model performance and efficiency. The formula for the pooling layer is as follows: (2)${\displaystyle y_{i,j,k}=ma{x}_{u=0}^{p-1}ma{x}_{v=0}^{p-1}{x}_{\left(i\times s+u\right),\left(j\times s+v\right),k}}$

where y_i,j,k represents the output of the k − th channel after pooling, s represents the step size, and p represents the size of the pooled area.

Fully connected layer

FCLs are pivotal elements within deep neural networks, facilitating highly flexible and expressive feature extraction and transformation (Li et al., 2023; Matsumura et al., 2023). They achieve this by establishing connections between every neuron from the preceding layer to each neuron in the subsequent layer, with each connection possessing an associated weight that governs the impact of the previous layer’s output on the current layer’s neurons.

FCLs exhibit numerous merits. Firstly, they boast potent expressive capabilities, enabling them to discern complex patterns and capture nonlinear relationships within the input data, thereby extracting essential abstract features. Secondly, FCLs offer remarkable flexibility, allowing them to adapt to diverse complex tasks and data distributions, thereby affording a considerable degree of model freedom. Additionally, FCL computations are straightforward, facile to implement, and amenable to acceleration through efficient matrix operations.

The roles of FCLs can be succinctly delineated as follows: initially, they transmute the original input data into higher-level abstract features by learning weights and biases, thereby extracting critical information. Subsequently, FCLs are commonly deployed in the final layer of neural networks to map the extracted features to specific classes or value ranges, thereby facilitating classification and regression tasks. Finally, FCLs undertake comprehensive evaluations on the input data and formulate decisions and predictions based on the acquired weights.

The formula for the full connection layer is as follows: (3)${\displaystyle y=f\left(Wx+b\right)}$

where y is the output vector, x is the input vector, W is the weight matrix, b is the offset vector, and f is the activation function.

One-hot coding

The NSL-KDD dataset comprises both numerical and categorical variables, yet deep learning algorithms inherently handle only numerical data, necessitating specialized techniques for managing categorical variables. Here, we delve into the process of independent one-hot encoding, exemplified by the character feature “protocol_type” (protocol type):

One-hot encoding is a method that converts categorical variables into digital formats amenable to machine learning algorithms. Essentially, it treats each possible value as a distinct binary feature. For a given data point, the feature corresponding to its value is assigned a value of 1, while features for other values are set to 0. The objective is to transform original text-based categorical features into binary sparse vectors, enabling deep learning algorithms to interpret and utilize them effectively.

In this context, there exist three distinct protocol types: “tcp”, “udp”, and “icmp.” After one-hot encoding, the “protocol_type” feature undergoes transformation into a sparse vector with a dimension equal to the number of protocol types, which is 3. This vector represents the protocol type utilized in each example, as illustrated in Table 2. Each column corresponds to a specific protocol type, such as “tcp”, “udp”, or “icmp”. If a particular protocol is employed in an example, the corresponding column is marked as 1; otherwise, it is designated as 0. This encoding methodology enables deep learning models to comprehend and process categorical variables, thereby facilitating subsequent analysis and modeling endeavors.

Upon examination of Table 2, it becomes evident that the character features encompass numerous categories. Each category undergoes independent one-hot encoding, yielding a corresponding sparse vector. For example, the “service” feature encompasses 70 distinct categories, leading to the generation of 70-dimensional sparse vectors post one-hot encoding. Similarly, the “flag” feature comprises 11 different categories, resulting in 11-dimensional sparse vectors post one-hot encoding. Consequently, character-type discrete features can be expanded to a certain extent through one-hot encoding.

Furthermore, the NSL-KDD dataset encompasses various attack features that can be broadly categorized into four primary categories, each with its own subcategories. A detailed breakdown of these categories is presented in Table 3.

It can be seen from the table that there are 39 sub-attack type tags in NSL-KDD data set tags. In order to prevent dimension explosion caused by single hot coding, each sub-class is now divided into five categories, namely DOS, Probe, U2R, R2L and Normal. The specific division rules are shown in Table 3 above.

SMOTE resampling

In deep learning, the issue of imbalanced data can result in inadequate predictive performance of the model for certain categories. In the NSL-KDD dataset, some attack categories have a limited number of samples, leading to insufficient attention from the model during training. Even when the model is trained on a large-scale dataset, the influence of dominant samples may introduce bias towards the categories they represent. The specific scenario is illustrated in Fig. 3A.

Figure 3A and Table 4 show that the number of DOS attack samples (45927) is significantly higher than that of the other three attack types (52 −11656), with U2R attack having the lowest number of samples (52), making it difficult to discern even with magnification.

To combat the issue of imbalanced data in machine learning tasks, the Synthetic Minority Over-sampling Technique (SMOTE) is an effective data balancing technique. Imbalanced data refers to a scenario where there is a significant difference in the number of samples between different classes in the training set, with one class having far more samples than the other classes. In such cases, models tend to favor the majority class, resulting in insufficient recognition of the minority class.

SMOTE addresses this problem by generating synthetic samples of the minority class to balance the dataset. It interpolates the minority class samples and generates artificial samples that are similar but not identical to the original samples. These synthetic samples fill the gaps in the original data, improving the balance of the dataset by increasing the number of minority class samples.

Resampling data with SMOTE offers several benefits. Firstly, it can enhance model performance by improving the classifier’s ability to recognize the minority class. By increasing the number of minority class samples, the model can better learn the features and patterns of the minority class during training, reducing misclassifications and improving overall classifier performance.

Secondly, SMOTE resampling can mitigate classifier bias towards the majority class. Due to the excessive number of majority class samples, models tend to predict the majority class more frequently, overlooking important information from the minority class. By generating synthetic samples, SMOTE balances the sample sizes between different classes, reducing this bias and enabling the classifier to treat different classes more equally.

Furthermore, the synthetic samples produced by SMOTE retain the characteristics of the original samples while introducing a certain degree of diversity. This increases the diversity of the data, provides more training samples, and enhances the model’s generalization ability (Shrinidhi, Kaushik Jegannathan & Jeya, 2023).

The fundamental principle of SMOTE revolves around gaining a deep understanding of a few categories of samples and subsequently generating new samples based on these selected ones to augment the dataset. The specific steps involved in this operation are as follows: (1) Selection: Randomly choose a sample point, denoted as X, from the selected categories. (2) Neighbor calculation: Determine the k nearest neighbors of the sample point in the feature space using primarily the Euclidean distance algorithm. (3) Generation of new samples: For each neighbor, create one or more composite samples based on their dissimilarities. The attributes of the new sample are obtained through linear interpolation between the attributes of the original sample and those of the neighbor sample. (4) Repetition: Repeat the above process until a sufficient number of synthetic samples have been generated.

The primary objective of the SMOTE algorithm is to enhance the performance and accuracy of the model by rebalancing the class distribution within the dataset through the iterative execution of the aforementioned steps. The key functions of the SMOTE algorithm are as follows: (8)${\displaystyle X_{\mathrm{new}}=x+rand\left(0,1\right)\ast \left|x-xn\right|.}$

Figure 3B demonstrates the outcome after applying SMOTE resampling, and it appears that the sample quantities among different classes have achieved a relatively balanced distribution. It can be seen that an equilibrium state has been reached between the classes, and the number of samples has increased from 125,973 to 336,710.

Finally, after resampling the data, the data is normalized. Normalization of features is to scale the range of different features to the same scale or range in order to eliminate the deviation or adverse effects caused by the difference of eigenvalues. In this article, min-max normalization is adopted, and its specific formula is as follows (9)${\displaystyle {\mathrm{y}}_{min-max}=\frac{x-min\left(x\right)}{max\left(x\right)-min\left(x\right)}.}$

The normalized results are all positive numbers and the data is scaled or mapped to the interval (0,1). The final results are stored in Excel table to facilitate subsequent operation and analysis. This series of steps helps to improve model performance, especially when dealing with uneven data.

Parameter tuning

The experimental environment configuration is shown in Table 5.

The SA-BO-CNN model improves its performance and generalization ability by adjusting various parameters, including hyperparameters, network structure, and data augmentation. The goal of parameter adjustment is to enhance accuracy and effectiveness in specific tasks, prevent overfitting, and optimize the training process and generalization capability of the model. It is an iterative process that requires repeated experiments and adjustments based on the actual situation to find the best configuration.

There are two commonly used methods for parameter tuning: grid search and Bayesian optimization. Grid search exhaustively searches through a finite set of parameter combinations, but it can be time-consuming and prone to the curse of dimensionality. Therefore, this article primarily utilizes the Bayesian optimization method to adjust parameters such as the number and size of convolution kernels, pooling size, number of nodes in the fully connected layer, and learning rate.

The first parameter being tuned is the learning rate. If the learning rate is too high, the model’s parameters may oscillate or diverge along the gradient direction, whereas if it is too low, the training convergence will be slow and may even get trapped in local optima. In the tuning process, the learning rate is adjusted within the range of (0.001, 0.01), and cross-validation is employed to evaluate the model’s performance under different learning rates. The debugging process is illustrated in Fig. 4.

The second parameter being tuned is the size of the convolution kernel. In this article, the adjustment range is set to (3, 5). The size of the convolution kernel affects the degree of abstraction of features and the receptive field size of the neural network. If the selected convolution kernel is too small, the neural network may not effectively capture large-scale features in the image. Conversely, if the selected convolution kernel is too large, the neural network may become too sensitive to detail information, which can lead to overfitting problems. The debugging process is illustrated in Fig. 5.

The third parameter being tuned is the number of convolution kernels. The feature extraction ability of the neural network can be adjusted by changing the number of convolution kernels. A lower number of convolution kernels may limit the neural network’s ability to learn complex features, resulting in underfitting. On the other hand, a higher number of convolution kernels may increase the neural network’s expressive power, but it also increases the risk of overfitting. The debugging process is illustrated in Fig. 6.

The fourth parameter being tuned is the pooling window size of the Pooling Layer. Selecting a pooled window size that is too large can result in losing critical information and important features. On the other hand, if it is too small, the model may become susceptible to noise or overfitting. In this article, the adjustment range is set to (2, 4). The debugging process is illustrated in Fig. 7.

The fifth parameter being tuned is the number of nodes in the fully connected layer. The fully connected layer is typically used to flatten the output features from the preceding convolution or pooling layers and perform tasks like classification or regression. Modifying the number of nodes in the fully connected layer can influence the model’s complexity. The debugging process is depicted in Fig. 8.

The change of accuracy and loss rate of the model after 10 iterations of Bayesian optimization is shown in Fig. 9.

The model tuning results are presented in Table 6. Based on the table, it is evident that this CNN-based model achieves an impressive accuracy of 98.36%. This accomplishment is attributed to the utilization of a 3 ×3 convolutional kernel with a count of 40, a 3 ×3 convolution kernel with a count of 61, pooling windows of size 3 ×3 and 3 ×3, a learning rate of 0.000553, along with fully connected layer nodes of 41, respectively.

Model validation and analysis

In this article, we mainly use Precision, Recall and F1-score to evaluate the performance of the model.

For a multi-classification problem, it is often necessary to number the categories, have K different categories, and express the i-th category as C_i. At the same time, suppose there are N_j samples with the true class label of C_j, among which $j\in \left[1,k\right]$. The classifier assigns M_i samples to class C_i, of which, $i\in \left[1,k\right]$.

Then the formulas of accuracy rate, recall rate and F1 score under multi-classification problem are as follows:

Precision rate: (10)${\displaystyle Precision=\frac{M_i}{\sum _{i=1}^k{M}_i}.}$

Accuracy rate measures the proportion of the samples correctly classified by the classifier in all the samples classified into this category.

Recall rate (Recall): (11)${\displaystyle Recall=\frac{M_i}{N_j}.}$

The recall rate measures the proportion of samples that the classifier can correctly classify in the real category.

F1 score: (12)${\displaystyle F1=\frac{2\times Precision\times Recall}{Precision+\mathrm{Re}call}.}$

F1 score is a comprehensive index of precision rate and recall rate, which weights and averages precision rate and recall rate, and obtains the maximum value when precision rate and recall rate are equal.

Under the condition of optimal parameters, the corresponding evaluation results of this article are shown in Table 7.

Based on Table 7, our SA-BO-CNN model demonstrates superior performance compared to other models. The model reported by Jiang et al. (2020) lacks a parameter tuning function, leading to an over-reliance on parameter selection and subsequently yielding a low precision rate. In addition, the study by Oluwakemi, Muhammad & Anyachebelu (2023) has undertaken more comprehensive efforts. They evaluate the efficacy of three distinct machine learning algorithms—CNN, recurrent neural networks (RNN), and Naive Bayes—in identifying diverse attack categories. The result indicates that CNN and RNN slightly outperform the naive Bayesian algorithms. The moderate performance is attributed to inherent limitations such as sensitivity to data noise and inability to process missing data. In Gan et al. (2022), a method combining a gradient coordination mechanism and focus loss is proposed exhibiting high accuracy. Nevertheless, it demands extensive parameter adjustments, posing a challenge in parameter tuning. Therefore, achieving improved results entails conducting numerous experiments to ascertain the optimal parameter configuration.