STB: synthetic minority oversampling technique for tree-boosting models for imbalanced datasets of intrusion detection systems

Experiment

In this experiment, the dataset that we use is an open dataset, which in detail describes the real condition of a network that has an IDS. The field in the dataset with the name KDDCup99 is available at the link: (https://www.kaggle.com/datasets/galaxyh/kdd-cup-1999-data), CICIDS 2017 is available at the link: (https://www.unb.ca/cic/datasets/ids-2017.html), and UNSW NB15 is available at the link: (https://research.unsw.edu.au/projects/unsw-nb15-dataset), describes the conditions of the attack which we mapped into parts of the attack on IDS. As for this dataset, we make dataset labels as in Tables 1, 2 and 3. We provide labels for each attack name at the label encoding stage in the pre-processing process. Data pre-processing is the initial technique of machine learning and data mining to convert raw data or commonly referred to as raw data collected from various sources into cleaner information and use it for further processing. This process can also be described as the first step in gathering all available information by cleaning, filtering, and combining this data. Data pre-processing is very important as errors, redundancies, missing values, and inconsistent data lead to reduced accuracy of analysis results.

Tables 1, 2 and 3 demonstrate the description of the dataset used in this experiment. In the original dataset, it appears that there is an imbalanced dataset that occurs. To deal with this problem, SMOTE is used and the number of datasets becomes balanced. Furthermore, each dataset is split into several training and testing datasets with a proportion of 80:20.

Normalization

Data normalization is the basic element of machine learning and data mining to ensure the record stays consistent in the data set. In this study, we changed the name of the column to a uniform form of initialization, such as the columns \“Flow Duration\”, \“Total Fwd Packets\”, \“Total Backward Packets\”, etc., we changed it to F1, F2, F3, etc. In the normalization process, data transformation is required or converts original data into a format that allows for efficient data processing. The main purpose of data normalization is to eliminate data redundancy (repetition) and standardize information for better data workflows. Data normalization is used to spread an attribute’s data to fall within a smaller range, e.g., −1 to 1 or 0 to 1. This is generally useful for classification algorithms. Data normalization techniques are very helpful as they offer many advantages as follows:

• The application of machine learning and data mining algorithms is easier

• Machine learning and data mining algorithms become more effective and efficient

• Data can be extracted from the database faster

• Normalized data can be analyzed by specific methods

Data cleaning

Data cleansing or data cleansing is one of the steps in data pre-processing. The purpose of this cleansing data is to select and eliminate data that has the potential to reduce the accuracy of machine learning and artificial intelligence. At this stage, we need to overcome the problematic data on our IDS dataset. In this study, some values are missing from the KDD Cup 99 dataset, so we manually filled in the missing data by averaging between above and below the missing value data. With the exception of the CICIDS 2017 and UNSW NB15 datasets, these datasets do not have a missing value. Some common problems encountered in datasets are as follows:

• Missing value when a value is missing from the record. For example, in a row table’s data, there is a cell with no value. To get around this, we replace the missing value with the median method. For this purpose, the mean value is determined and missing values are replaced by the mean value.

• Noisy data when the data contains incorrect or anomalous values. The condition is also known as an outlier. In this study the KDD Cup 99, CICIDS 2017, and UNSW NB15 datasets had noisy data, these datasets had problems with inaccurate labeling and a large number of duplicates. To overcome noisy data, there are several techniques that we performed, including:

  • Binning, a method we use that divides data into multiple partitions and then treats the partitions separately. Then the mean, median or specified limit value is searched from all data partitions.

  • Regression, a method we use that uses the linear regression equation to predict the value of data. This method can be used when there is only one independent attribute.

  • Clustering, a method we use to create a group or cluster of data of similar value. Values that don’t get into the cluster can be considered noisy data.

• Inconsistent data, which is the condition when the values in the data are inconsistent. In this study, the data from KDD Cup 99, CICIDS 2017 and UNSW NB 15 showed inconsistent data. To overcome this inconsistent data, we use noisy data, binning, regression and clustering methods.

Label encoding

Label encoding is a pre-processing of IDS dataset where we try to change the data type of the categorical column to numeric (from string to numeric). The columns we changed include: For the KDD Cup 99 record, the records in the Protocol_type, service, and Flag columns have string values like tcp, http, sf, etc. We change them to the numbers 1, 2, 3, etc. For UNSW NB 15 records, in the Protocol, Service, and Status columns, the records have string values like tcp, ftp, and FIN, etc., we change them to the numbers 1, 2, 3, etc. Meanwhile contains the CICIDS 2017 dataset doesn’t have a string value in each dataset, so we won’t change anything. This happens because the machine learning model does not understand the character of the string, and therefore a provision must be made to convey it in a format that the machine learning can understand. This is achieved through the coding label method. In the tagging coding method, the category under the categorical feature is changed in a manner involving hierarchical separation. That is, if we have categorical features in which the categorical variables are hierarchically related to each other, then we need to label those features. When label-encoding is performed on non-hierarchical features, the accuracy of the model is severely compromised and is therefore not a good choice for non-hierarchical features. To see more detailed dataset labels, please refer to Tables 1, 2 and 3.

SMOTE for imbalanced dataset

Imbalanced data is a situation where the target class or category has a significantly different frequency in the training data. This means that one or more classes are under-represented in the data set, while one or more classes are over-represented. In this study, the datasets we used had imbalanced classes, including: In the KDD Cup 99 dataset, some classes in the dataset had a much larger number of samples than others, with the class representing the attack having a much smaller number of samples compared to normal traffic classes. In the CICIDS 2017 and UNSW NB 15 datasets, some classes have attack frequencies, anomalous representation, and consistency that differ from normal classes. This data set also contains duplicate and irrelevant data. Imbalanced data can cause problems with machine learning models as they can be biased toward the majority class and have trouble spotting the minority class. This can lead to poor performance metrics for minority classes, such as accuracy, precision recall and F1 Score. To overcome this problem, the researcher proposes first determining the nearest majority data point from the sample selection and then replacing the nearest neighbour parameter with a safe radius distance. Then the safety radius becomes a reference in creating synthetic stone data points. The circle formula with a two-dimensional vector is used to generate data as shown in formula (1).

(1)${\displaystyle \parallel \underset{\sim }{b}-p\parallel \le r^2}$ (2)${\displaystyle \sum _{i=1}^n{\left(b_{ij}-P_{ij}\right)}^2\le r^2}$ (3)${\displaystyle r^2=\sum _{j=1}^n{\left(p_j-t_j\right)}^2}$ Where the point “p” is the minority sample point which is the center of the circle with $\left(p_1,p_2,p_3,\dots ,p_n\right)$ as in formula (2). Point “b” is a new generation synthetic point of interpolation between the two directions with $\left(b_1,b_2,b_3,\dots ,b_n\right)$ as in formula (2). Point “t” is the majority point closest to the center of the circle with $\left(t_1,t_2,t_3,\dots ,t_n\right)$ as in formula (3). On the other hand, r² is the distance between p and t can in formula (3).

The next step is to perform calculations to generate new synthetic data. Using Euclidean formula, the distance between each majority data and each minority sample is calculated. The smallest distance from the entire selected minority sample is the majority data point. As shown in formula (4). (4)${\displaystyle r_{ij}=min\sum _{i=1}^n\sum _{j=1}^n\sqrt{{\left(p_j-t_i\right)}^2}}$ where r_ij is the smallest distance between majority sample i and minority sample j. The generation of new synthetic data is carried out on the interpolation line between the next predetermined majority data points and the minority data sample points. The new generation of synthetic data based on the safe radius distance from the first direction interpolation scheme is categorized as a_i with $\left(a_1,a_2,a_3,\dots ,b_n\right)$. In addition, both line directions are used to generate synthetic r_ij and −r_ij, which can be seen in formula (5) and (6).

(5)${\displaystyle a_{ij}=P_j+\left(\text{rand}0,1\times \left(r_{ij}-p_j\right)\right)}$ (6)${\displaystyle b_{ij}=P_j+\left(\text{rand}0,1\times \left(p_j-r_{ij}\right)\right).}$

The restricted area for creating new synthetic data is then used to reduce the occurrence of overlapping data in the SMOTE process.

The Synthetic Minority Oversampling Technique (SMOTE),

Splitting data

In this study, the first dataset used was KDDCup99 with a total number of records are 494,020, then the CICIDS 2017 dataset with a total number of records are 56,661 and finally UNSW NB15 with a total number of records are 540,044. With this split data, the data from each dataset is divided into two, namely training data and test data, with a ratio of 80% for training data and 20% for test data. Split data is a more efficient option because it can split Intrusion Detection System (IDS) datasets into training subsets and test subsets quickly compared to cross validation because it will take a long time. The use of cross validation will cause the results of model evaluation to vary depending on how the data is divided into different subsets. This will lead to a degree of dependence on the data sharing performed. We want to avoid this dependency and use the data split method to get more consistent results. After dividing the two data into training data and test data, the amount of data is: For the KDD Cup 99 dataset, the number of training data is 395,216 and the test data is 98,804. For CICIDS 2017, the number of training data is 45,328 and the test data is 11,333. For the UNSW NB 15 dataset, the training dataset is 108,009 and the test dataset is 432,035.

Apply the machine learning models

In this research, we use three-boosting machine learning models including: LightGBM, XGBoost and CatBoost as explained in the previous chapter. These three-boosting machine learning has its own unique characteristics and learning approach. By using three different algorithms, we introduce diversity in the models being trained. This helps to capture different aspects of the imbalanced dataset and potential patterns associated with intrusive instances. The combination of multiple models can lead to more robust and accurate predictions. Parameters we use in each model which can be seen below.

LightGBM

LightGBM is a gradient boosting framework that is efficient and distributed, providing faster training speed, greater efficiency, less memory usage, and better accuracy. It is a decision tree-based machine learning algorithm that competes with XGBoost, which used to dominate Kaggle competitions. LightGBM is now more popular among Kagglers because of its high speed, accuracy, and ability to handle large amounts of data. It is also known for its GPU learning support and its focus on accuracy. However, LightGBM is not advisable for small datasets and is sensitive to overfitting. In this research of the LightGBM model, the number of leaves per tree we use is 63, the speed iteration control is 0.01, specifies the fraction of data to use for each iteration, and is generally used to guide training speed up and avoid overfitting with a value of 0.5. The proportion of features is randomly chosen in each iteration to create trees with a value of 0.5.

XGBoost

XGBoost is a machine learning algorithm used for regression, classification and ranking problems. It’s open source and efficient for large and complex datasets. XGBoost can handle missing values and function interactions, has built-in regularization techniques, and works by building a series of decision trees that correct the errors of previous trees. A popular choice among data scientists, XGBoost is used in various applications such as financial modelling, healthcare, and natural language processing. In the XGBoost model, n_estimators, which determine the epoch of the model, is set to 100 and early_stopping_rounds to 10 to check for overfitting. The grid values sought for the learning rate is 0.01. The max_depth is 4 and the min_child_weight in selected ranges from 1 to 10.

CatBoost

CatBoost is an open-source gradient boosting library used for classification, regression, and ranking tasks. It handles categorical variables better than other algorithms, making it useful for datasets with a mix of numeric and categorical data. It uses ordered boosting to process categorical variables and handles categorical variables with high cardinality more effectively. CatBoost can handle missing values without any special pre-processing, saving time and effort for data scientists. It is powerful, efficient and used in various applications like web search ranking, recommender systems and computer vision. In the CatBoost, the grid values sought for the learning rate is 0.01. The iteration is 100 and the maximum depth of the tree is 10. It is capable of handling model overfitting and the number of l2_leaf_reg per tree we use is 1.

Evaluation metrics

To evaluate the performance of LightGBM, XGBoost, and CatBoost on the three datasets we use, the metrics we use are as follows:

(7)${\displaystyle \text{Precision}=TP/\left(TP+FP\right)}$ (8)${\displaystyle \text{Recall}=TP/\left(TP+FN\right)}$ (9)${\displaystyle \text{Accuracy}=\left(TP+TN\right)/\left(TP+TN+FP+FN\right)}$ (10)${\displaystyle \text{F1-Score}=\left(2\times \text{Precision}\times \text{Recall}\right)/\left(\text{Precision}+\text{Recall}\right).}$

Precision is the ratio of positive correct predictions compared to the overall positive predicted results in the IDS dataset used. Recall is the ratio of correctly positive predictions compared to all the correctly positive data. Is the ratio of correct predictions (positive and negative) with all the IDS data that we use. The F1-score is a comparison of the average precision and recall which is weighted. Finally, accuracy answers the question “how accurately are the predictions of the types of attacks processed by LightGBM, XGBoost, and CatBoost”.