A comparative study of dimensionality reduction techniques for intrusion detection in IoT networks

Prediction models

A wide variety of ML and deep learning (DL) models have been proposed for the development of IDSs, including SVMs, logistic regression, decision trees or NNs, just to mention a few. Although DL approaches often achieve high performance, they are typically too resource-intensive for resource-constrained environments, even after applying aggressive optimisations (Daghero et al., 2021). Consequently, this work focuses on RF models, an ensemble method that builds multiple decision trees using different subsets of the data and combines their outputs to produce more accurate and stable predictions.

RF models are considered a standard in ML literature due to their flexibility, strong classification performance, and inherent interpretability (Bakro et al., 2024). This last aspect is particularly relevant in the context of cybersecurity, where understanding why an alert has been raised is often as important as the alert itself. Moreover, they require minimal parameter tuning, which makes them easy to implement for practitioners working in real-world environments.

It should be noted that the computational cost of RF models grows with the amount of trees in the ensemble. This is particularly relevant when deploying models on resource-constrained IoT devices. Therefore, the number of trees in all RF models has been set to a moderate value of 20 throughout this work.

Dimensionality reduction methods

Dimensionality reduction encompasses two fundamental approaches: feature selection, which consists of identifying and retaining the most relevant variables from the original dataset; and feature extraction, which involves transforming the original features into a new set of variables that capture the essential structure of the data.

FS techniques can be further classified into filter and wrapper approaches (Chen et al., 2020). The former consist of scoring the relationship between each feature and the target variable according to a specific criterion, and discarding those with the lowest scores. The number of features to retain is determined by a user-specified threshold, which is usually a fixed number of features to keep or a minimum score below which features are excluded. On the other hand, wrapper methods assess feature subsets by training a predictive model and using its performance to guide the selection process. These approaches are usually more accurate than filter methods, since they can detect interactions between features. However, these strategies entail a higher computational cost, since they require training and validating a model multiple times across different feature subsets (Morán et al., 2024).

In contrast to FS, FE techniques reduce dimensionality by transforming the original variables into a new set of features. Their main drawback is the loss of interpretability, as the new features typically lack intuitive meaning within the original problem domain. Furthermore, when deploying an IDS on resource-constrained IoT devices, FE requires on-device transformation, thereby increasing the computational load during inference. One of the most common FE methods is PCA, which consists of transforming the original feature set into a new set of uncorrelated variables known as principal components, ordered by the amount of variance they capture.

To provide a comprehensive comparison, all the approaches described above need to be considered. Therefore, this analysis includes three widely used filter-based feature selection techniques —the ${\chi }^2$ test, Mutual Information, and Permutation Feature Importance—, one wrapper method (Genetic Algorithm), and one feature extraction technique (PCA). A brief description of each method is provided below.

• 1.

  ${\chi }^2$ test: the ${\chi }^2$ test is an statistical technique that evaluates the dependency between each feature and the target class by measuring how the observed distribution of values differs from the expected one. Features with higher ${\chi }^2$ scores are considered more relevant for classification (Fatima, Rehman & Rehman, 2023).

  It should be noted that the ${\chi }^2$ test is designed for categorical or nominal variables only (Morán et al., 2024). When applied to continuous features, the test treats each unique value as a separate category. In this case, it does not return an error as long as the feature values are non-negative, but this misinterpretation may lead to biased results: features with high variability can appear artificially significant due to their large number of unique values. Despite this limitation, it is common in the cybersecurity literature to apply the ${\chi }^2$ test directly to continuous features without any prior discretisation. For instance, in recent works (Nururrahmah & Ahmad, 2023; Fatima, Rehman & Rehman, 2023; Kilichev, Turimov & Kim, 2024), the ${\chi }^2$ test was employed for IDS development using datasets such as KDD Cup 99, NSL-KDD, ToN-IoT, and Edge-IIoTset without reporting any specific preprocessing for continuous variables. This tendency may be attributed either to an implicit reliance on standard tools to manage these issues internally, or simply to the fact that, in many cases, satisfactory results are achieved, leading authors to overlook the potential bias introduced. In light of the discussion above, and given its common use in the literature, it was decided to include the naïve ${\chi }^2$ method in this article to compare its performance against the other approaches.

• 2.

  Mutual Information (MI): MI is a symmetric, non-negative value that indicates the dependency between two random variables X and Y by measuring the reduction of uncertainty about the former having observed the other (Altmann et al., 2010). Formally, MI measures the divergence between the joint distribution $P(X,Y)$ and the product of the marginal distributions $P(X)P(Y)$ as follows: (1) $$MI(X,Y)=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{P}_{XY}(x,y)\log \frac{P_{XY}(x,y)}{P_X(x)P_Y(y)}$$where $\mathcal{X}$ and $\mathcal{Y}$ represent the probability spaces over which the random variables X and Y are defined.

• 3.

  Permutation Feature Importance (PFI): in decision tree-based models such as RF, FI is commonly computed by measuring how much a feature contributes to reducing node impurity, typically using the Gini index criterion, when it is selected for a split (Strobl et al., 2007). Alternatively, permutation-based FI evaluates how much the performance of the model degrades when the values of a given feature are randomly permuted. If the shuffling process leads to an increase in prediction error, it indicates that the model depends on that feature to make accurate predictions (Molnar, 2022). In this article, the latter approach has been chosen, since it has been reported that the former tends to introduce bias in multiclass prediction models (Altmann et al., 2010).

• 4.

  Principal Component Analysis (PCA): a projection-based technique that converts the original set of features into a new set of uncorrelated variables known as principal components. Each component is formed as a linear combination of the original variables, and are ordered according to the amount of variance they capture from the original data. The number of components selected to fit a model is usually determined by setting an explained variance threshold, typically around 95% to 99% (Alhowaide, Alsmadi & Tang, 2020).

• 5.

  Genetic Algorithm (GA): Metaheuristic algorithms are optimization techniques designed to efficiently explore large and complex search spaces, often inspired by natural or physical processes. In the context of DR, metaheuristic strategies have gained significant attention due to their flexibility and effectiveness for FS (see, for example Fatani et al. (2021a, 2021b), Singh & Ujjwal (2023)). In this work, a GA has been selected as a representative example of the application of metaheuristic approaches for DR.

  Within the GA framework, each candidate subset of features is represented as a binary vector whose length corresponds to the total number of predictors. After randomly generating an initial population of candidate feature sets, the algorithm uses evolutionary strategies such as mutation and crossover to identify subsets that optimise a given performance metric, such as classification accuracy (Halim et al., 2021). The main limitation of this approach lies in its computational burden, as selecting features by means of GA requires fitting hundreds of models.

Dataset description

Selecting adequate datasets is essential when designing AI-based IDSs. However, most IoT cybersecurity works utilise generic datasets that fail to reflect the specific characteristics of IoT traffic. According to the review by Aversano et al. (2021) the most commonly employed datasets are NSL-KDD, UNSW-NB15, and KDD Cup 99, none of which were specifically designed for real IoT scenarios. Furthermore, it is important to ensure diversity in both data volume and attack types to conduct a robust analysis. For instance, the BoT-IoT dataset provides a large amount of data but includes only four types of threats, namely: reconaissanse, theft, DoS and DDoS (Koroniotis et al., 2019). Taking these considerations into account, the numerical experiments have been conducted on two publicly available benchmark datasets developed specifically for IoT intrusion detection: Edge-IIoTset (https://www.kaggle.com/datasets/mohamedamineferrag/edgeiiotset-cyber-security-dataset-of-iot-iiot) and CICIoT2023 (https://www.kaggle.com/datasets/akashdogra/cic-iot-2023).

The Edge-IIoT dataset (Ferrag et al., 2021) provides a comprehensive IoT traffic setting, contains a broad range of attacks, and includes a total of 61 selected features describing various aspects of the network. A summary of the attack scenarios, their effects, and the most relevant features for their characterisation, according to expert knowledge, is presented in Table 1. Note that the source and destination IP addresses have been explicitly omitted from the features column. Both variables can be relevant when detecting attacks since, for example, if a command-and-control has been installed on a device, it will connect to an unknown external IP address. However, considering these features can undermine the generality of the models, introducing a bias towards a blacklist of malicious IP addresses.

On the other hand, the CICIoT2023 dataset (Neto et al., 2023) contains network traffic captured in a testbed composed of 105 real IoT devices, including a variety of brands and types, providing an accurate representation of a real-world deployment in a smart home environment. Of particular relevance is the use of IoT devices as malicious agents, reflecting realistic internal threat scenarios where infected nodes can initiate attacks against other IoT devices. In addition to benign traffic, the dataset includes a total of 33 distinct attack types, grouped into seven main categories: Denial of Service (DoS), Distributed Denial of Service (DDoS), brute force, spoofing, reconnaissance, web-based attacks, and Mirai-related attacks. The main challenge when working with the CICIoT2023 dataset is its massive size, containing over 46 million records and 47 features per instance, including the label feature that identifies the type of attack. Consequently, processing this dataset requires careful data management, which is described below.

Dataset preprocessing

Since the data in the Edge-IIoTset and CICIoT2023 datasets originate from traffic captured in real testbed environments, a preprocessing phase is necessary before training the prediction model. For this analysis, the following preprocessing steps were applied:

• 1.

  Dataset merging: Due to its large scale, the CICIoT2023 dataset is distributed across 169 separate CSV files, each containing a portion of the overall traffic. All files were merged into a single file to allow model training.

• 2.

  Removing unnecessary columns: following the recommendations of Ferrag et al. (2022), the authors of the Edge-IIoTset dataset, features identified as irrelevant for the analysis or potentially introducing bias into the model were excluded. For instance, as noted above, features like ip.src_host, ip.dst_host, and the ARP protocol equivalents directly identify the source and destination devices in the network, enabling the classifier to memorise device identifiers rather than detecting anomalies in network behaviour. Similarly, frame.time must be excluded because if the attacks were carried out at a specific time, the model could associate certain timestamps with them, instead of learning network patterns. As an additional example, the mqtt.msg feature can include application-level data such as sensor readings or textual content that may directly reveal the attack class. Consequently, it could bias the model towards recognising specific payload patterns instead of protocol-level anomalies. The full set of excluded variables can be found in the code repository associated with this article.

• 3.

  Removing constant columns: features with a single unique value across all records were removed, as they do not contribute to the model.

• 4.

  Encoding categorical features: non-numeric features were transformed into numerical format following an ordinal encoding scheme.

• 5.

  Handling duplicates and missing values: duplicate entries and records containing missing values were detected and eliminated. This step was particularly important in the CICIoT2023 dataset, where more than 22 million rows, representing almost half of the total, were removed.

• 6.

  Detecting and removing identical columns: redundant features containing identical information were discarded.

• 7.

  Data scaling: continuous features were log-transformed to reduce skewness and limit the impact of extreme values. This step is of particular importance for the PCA method, given its sensitivity to feature variance and magnitude. Specifically, PCA requires feature scaling, typically through standardisation. However, when features are highly skewed toward large values, standardisation alone may lead to distortions in variance, causing certain variables to dominate the principal components. The logarithmic transformation before standardisation mitigates this effect, improving the performance of the PCA models.

• 8.

  Undersampling. This preprocessing step was applied only to the CICIoT2023 dataset, due to its significantly larger size, even after removing duplicate records. Moreover, CICIoT2023 dataset is extremely imbalanced, with classes containing several millions of records while others only a few thousands. To address both issues, a simple yet sensible rule-based undersampling strategy was implemented: those classes with fewer than 100,000 samples were preserved entirely, those with more than 100,000 but fewer than 500,000 were reduced to 60% of their original size, from 500,000 to 5 million they were reduced to 40%, and the extremely large class DDoS-ICMP_Flood, containing more than seven million records, was limited to 20% of its instances.

• 9.

  Data splitting: for training and testing purposes, both datasets were split into proportions of 80/20, with stratification by attack type to preserve the class distribution across the training and test sets.

Experimental setup

To provide a fair comparison between the different DR methods, three feature rankings were first constructed based on the ${\chi }^2$ test, MI, and PFI criteria described in ‘Dimensionality Reduction Methods’. Similarly, PCA was applied to compute the set of principal components from the original feature space. Subsequently, an incremental evaluation was conducted for each method using a sequential forward selection strategy: at each step, one additional feature (or principal component), selected according to the corresponding ranking, was included in the RF model fitting process. Each model was constructed and evaluated in terms of accuracy, model size, and inference time following a hold-out validation scheme, with the training and testing sets specified in ‘Dataset Preprocessing’. In parallel, a Genetic Algorithm was employed on the training set to identify feature subsets that maximise model accuracy, which was estimated using cross-validation, and the resulting model was assessed on the test set using the same metrics. To reduce variability in the inference time measurements, the prediction step was repeated ten times on the test set, and the median value of all runs was recorded as the model inference time.

After completing this process, the selection of the best model produced by each method was based on its predictive power on the test set, determined by accuracy. Among the top-performing models, those that made the most efficient use of computational resources were preferred. Once the best models were identified, a more detailed evaluation of them was conducted using the precision, recall, and F1-score metrics defined in Eq. (2). Finally, the best models were deployed on a Raspberry Pi device, and assessed in terms of inference time and RAM usage during the prediction stage.

(2) $$\begin{array}{rl}\mathbf{A}\mathbf{c}\mathbf{c}\mathbf{u}\mathbf{r}\mathbf{a}\mathbf{c}\mathbf{y}=\frac{TP+TN}{TP+TN+FP+FN}\hfill & \mathbf{P}\mathbf{r}\mathbf{e}\mathbf{c}\mathbf{i}\mathbf{s}\mathbf{i}\mathbf{o}\mathbf{n}=\frac{TP}{TP+FP}\\ \mathbf{F}\mathbf{1}\text{−}\mathbf{s}\mathbf{c}\mathbf{o}\mathbf{r}\mathbf{e}=2\cdot \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\cdot \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}\hfill & \mathbf{R}\mathbf{e}\mathbf{c}\mathbf{a}\mathbf{l}\mathbf{l}=\frac{TP}{TP+FN}\end{array}.$$

The original training and testing of models were conducted on high-performance machine equipped with 2x Intel Xeon Silver 4210 CPUs (10 cores, 2.2 GHz) and 96 GB of RAM, running Python 3.13.1, with Conda 24.11.2, Scikit-learn v1.4.2, Pandas v2.2.3, and NumPy v1.26.4. On the other hand, the resource-constrained Raspberry Pi device was equipped with an ARM Cortex-A72 processor (quad-core, 64-bit, 1.8 GHz) and 8 GB of RAM, running Python 3.10.16, with Conda 25.1.1, Scikit-learn v1.6.1, Pandas v2.2.3, and NumPy v1.24.3.

Edge-IIoTset

This section presents the numerical results obtained by applying the techniques described in ‘Experimental Setup’ to the Edge-IIoTset dataset. Figure 1 illustrates that increasing the number of features included in the model leads to a convergence in accuracy among all methods, reaching approximately 98.9% in each case. On the other hand, Figs. 2 and 3 depict the corresponding prediction time model size for each approach, respectively. To facilitate a more detailed comparison, the selected models obtained by each method, including the one obtained via GA, are outlined in Table 2.

The experiment demonstrates that models with similar performance in accuracy can differ significantly in terms of computational efficiency. It must be emphasised that the goal of DR should not be the minimisation of the final number of features, but rather to improve the predictive performance of the model to optimise the use of computational resources. In this light, Fig. 2 shows that, in general, inference time tends to increase slightly as more features are included in the model. Notable exceptions are the models obtained through PCA and the ${\chi }^2$ test. In the case of PCA, models trained with only one or two components exhibit very high inference times, although this stabilises at lower values as more components are added. More interesting is the case of the ${\chi }^2$ test, where the prediction time increases dramatically with the inclusion of the fifth feature, and then gradually decreases as more variables are incorporated to the model.

This unusual behaviour can be explained by the fact that the decision trees were fully grown and left unpruned. When the feature set is made up of features without enough discriminative power or includes contradictory patterns, the trees tend to grow deeper to separate the data, leading to a substantial increase in model complexity. As more features are considered, trees can find more useful and faster splits, reducing the need to grow as much. As a result, the trees are smaller, and the final model becomes lighter, even with more variables. This phenomenon exemplify the existence of ill-posed feature combinations, in the sense that they introduce instability or noise into the learning process, leading to unnecessarily complex models. Therefore, this finding suggests the existence of well-posed feature subsets capable of minimising resource demands while preserving predictive capability.

With respect to model size, Fig. 3 shows how it evolves as the dimensionality increases. The results are consistent with those observed for inference time, highlighting the pronounced increase in model size for ${\chi }^2$-selected subsets before stabilising. Notably, while the ${\chi }^2$ test, Mutual Information (MI), and PCA methods produce models whose sizes stabilise around the baseline (85 MB), the models trained on subsets selected by PFI require only around 47 MB, resulting in a storage saving of approximately 45%.

Finally, as shown in Table 2, the feature subset obtained through Genetic Algorithm produces a model that is both compact and highly accurate, but slow in terms of inference time. This outcome is expected, as the objective of GA-based feature selection is to maximise accuracy without taking into account computational efficiency.

CICIoT2023

This section presents the results obtained on the CICIoT2023 dataset. In line with ‘Edge-IIoTset’, Figs. 4, 5 and 6 show the evolution of model performance for the ${\chi }^2$-test, MI, PFI and PCA approaches in terms of accuracy, inference time, and model size, respectively. Similarly, Table 3 provides a more detailed comparison of the selected models. Overall, the results obtained are consistent with those observed for the Edge-IIoTset dataset. However, there are some relevant differences worthy of closer examination.

The most notable discrepancy is the poor performance of the PCA method. PCA-based models exhibit a substantial decrease in accuracy compared to the other approaches, while requiring twice the inference time and four times more storage. Furthermore, due to the high computational cost, it was not possible to train models using only one or two principal components, even when using high-performance hardware.

A possible explanation for the poor performance of PCA could be the dominance of the Inter-Arrival Time (IAT) feature, which appears to provide most of the discriminative information. This is supported by the fact that both MI and PFI methods ranked IAT as the most relevant feature, leading to models that reach around 98% accuracy using IAT alone. In the same line, the ${\chi }^2$-test approach identified IAT as the sixth feature in relevance, and its inclusion led to a nearly 30-point gain in accuracy, along with substantial reductions in inference time and model size. Since PCA constructs components that capture global variance without regard to class labels, it tends to combine IAT with other features in the first few components. This transformation may dilute the discriminative power of IAT, resulting in the model failing to produce accurate results.

Beyond the specific behaviour of PCA, a general increase in model sizes and inference times was observed. This is to be expected, as CICIoT2023 is significantly larger and more complex than Edge-IIoTset. Furthermore, the test set, representing 20% of the dataset, was considerably larger in this experiment, naturally leading to longer inference times. Nonetheless, the relative performance of the different methods remained consistent with the results obtained for Edge-IIoTset. Once again, PFI produced the most balanced models in terms of predictive power and resource consumption. The GA also delivered satisfactory results, yielding the most accurate model while maintaining a low inference time.

Performance on Raspberry Pi

While the previous analyses focused on the predictive performance and computational requirements of the models in a high-performance environment, it is also important to evaluate their behaviour on resource-constrained devices, representative of real IoT networks. To this end, a Raspberry Pi was used to assess the practical feasibility of deploying the selected models. The evaluation focused on two key metrics: inference time and RAM usage. To ensure measurement stability, each model was executed ten times. Results are presented in Tables 4 and 5 for the Edge-IIoTset and the CICIoT2023 datasets, respectively. The reported values include the mean, median, standard deviation, and 95% confidence interval for both metrics. Given the large size of the CICIoT2023 test set, it was necessary to subsample it for use on the Raspberry Pi. Specifically, 10% of the set was retained using stratified sampling to preserve the original class distribution. Despite this consideration, due to its large memory footprint, the PCA-based model could not be executed on the Raspberry Pi for the CICIoT2023 dataset. In line with previous experiments, the models selected using the PFI were found to be the fastest and most efficient in terms of RAM usage. For the Edge-IIoTset dataset, inference time was reduced by approximately 8%, and RAM consumption decreased by about 5%. Interestingly, although the PCA model used slightly less RAM, it was the slowest to execute. This, combined with its lower accuracy, made the PFI model preferable. In the case of the CICIoT2023 dataset, the improvements were even more significant, with reductions of around 11% in inference time and 28% in RAM usage. This improvement is particularly relevant, as it represents nearly 1GB of memory savings on device equipped with only 8 GB of RAM. To strengthen the validity of these findings, ANOVA tests followed by multiple comparison analyses were conducted for both metrics and datasets. The results confirmed that the differences observed among considered methods were statistically significant, with p-values ranging from ${10}^{-3}$ to ${10}^{-7}$ in the multiple comparisons tests. Full details of the analysis are available in the public GitHub repository linked to this article. Finally, the selected models were significantly more compact, with model sizes reduced by 45% on Edge-IIoTset and by 18% on CICIoT2023, respectively. These improvements facilitate practical deployment in real-world IoT networks, where storage and memory constraints are often limiting factors.