Improved temporal IoT device identification using robust statistical features

Data collection phase

The data collection phase involves gathering relevant data to create a comprehensive dataset for analysis and modeling. Our article employs two datasets from different sources, as shown in Table 1, with their details as follows:

1. IoT Traffic Traces: A publicly available dataset from the University of New South Wales (UNSW) (Sivanathan et al., 2019). The IoT Traffic Traces includes network traffic from 23 IoT devices and seven non-IoT devices, collected over 60 days. The PCAP files are segregated based on each day.

2. IoT-FCSIT: Our dataset was collected over six weeks explicitly for this article’s purpose of assessing the proposed feature set’s robustness across various network environments. This dataset simulates the average consumer’s typical usage, such as the Smart Plug being turned on when entering the lab and off when leaving. It was gathered over six weeks in the research lab at the Faculty of Computer Science and Information Technology (FCSIT), Universiti Malaya (UM), and is available to access through DOI: 10.6084/m9.figshare.25143581.v1.

Data processing phase

The data processing phase involves preparing raw data for model training. The outcome is a processed dataset that is error-free, contains relevant features, and is appropriately structured for machine learning algorithms. The data processing phase encompasses four essential processes: data cleaning, feature selection, data transformation, and data splitting.

Figure 2 shows the data cleaning step where the raw data in a PCAP file is processed using the Pyshark library to transform it into a dataframe, which is then saved as a CSV file. Then, any data points with incomplete information are removed from the dataset to handle missing data. Additionally, non-IoT traffic, irrelevant to the current analysis, is eliminated from the dataset. Our article filters out non-IoT traffic based on MAC and IP addresses for the public dataset, IoT Traffic Traces. Meanwhile, for the private dataset, IoT-FSCIT, we configured our setup exclusively to ensure our dataset is free from any non-IoT traffic from the start, thereby eliminating the need for post-collection filtering.

The features implemented in this article were originally employed in our previous work on granular network traffic classification (Zaki et al., 2022). Our previous article aimed to classify network traffic based on inter-application and intra-application services. Inter-application services are similar services associated with different parent applications, like Facebook-comment and YouTube-comment. On the other hand, intra-application services encompass different services within the same parent application, for instance, YouTube-comment and YouTube-post. We introduced 52 initial payload-length-based features in the previous work, as documented in Table 2. We have extended the previous work by implementing these same features to effectively reduce accuracy degradation over time in IoT device identification systems.

Next, we compute Pearson’s correlation coefficient (r) to determine the most relevant features. Pearson’s correlation coefficient measures the statistical relationship and magnitude of the correlation between continuous variables. Features with low correlation values are less dependent on other variables and are thus more effective in diverse network environments. For this article, we selected features with only low degree (−0.29 ≤ r ≤ 0.29) and moderate degree (ranging from −0.49 ≤ r ≤  − 0.30 and 0.30 ≤ r ≤ 0.49) correlation values to boost their adaptability to different network environments. Considering the reliability of correlation values, we found that the feature max_payload consistently exhibits the lowest correlation values in both the public (IoT Traffic Traces) and our (IoT-FCSIT) datasets, which indicates its robustness in the context of Pearson correlation.

Additionally, instead of using 10, 20, 40, and 100 packets to enhance efficiency, we opt for a more efficient window size of five packets. A smaller window size allows quicker updates to the statistical feature values, making the feature set more responsive to changes in network traffic patterns. These findings contribute to developing an efficient IoT device identification model while reducing accuracy degradation over time. Table 3 lists the seven selected features, including each feature’s description and correlation coefficients.

The following equations describe all the selected features using statistical measures from the payload length. These measures help us gain insights into the characteristics of the network traffic.

Average payload length within a 1-second duration (avg_payload_1_sec)

The first feature, avg_payload_1_sec, quantifies the average payload length across a discrete 1-second interval. It entails aggregating the payload lengths of all packets within this temporal window and subsequently dividing this sum by the total count of packets (denoted as n) within this interval. Equation (1) shows the definition: (1)${\displaystyle \begin{array}{c}\hfill {{\mathrm{avg}}_{\mathrm{payload}}}_{1_{\sec }}=\frac{1}{n}\sum _{j=1}^n{P}_j\hfill \end{array}}$

where, n signifies the total number of packets within the 1-second interval, and P_j corresponds to the payload length of the j-th packet. Utilizing the 1-second duration is preferred due to its minimal time frame, as opposed to five or 10-second intervals. This choice ensures a more rapid response to changes in network traffic and expedites the feature extraction process.

Statistical measures employing a 5-moving sliding window.

This suite of features (i.e., sum, mean, variance, and standard deviation) leverages a 5-moving sliding window technique to analyze the payload lengths. Let P_i denote the payload length of the i-th packet and the 5-moving sliding window is computed for each packet from the current packet, i, to the four subsequent packets (i + 4). Equation (2) shows the summation of payload lengths within a 5-moving sliding window: (2)${\displaystyle \begin{array}{c}\hfill {\mathrm{fma5}}_{\mathrm{sum}}=\sum _{j=1}^5{P}_i.\hfill \end{array}}$

The mean payload length within this window is calculated by dividing the sum of the payload lengths by the window size, as shown in Eq. (3): (3)${\displaystyle \begin{array}{c}\hfill {\mathrm{fma5}}_{\mathrm{mean}}=\frac{1}{5}{{\mathrm{fma5}}_{\mathrm{sum}}}_i.\hfill \end{array}}$

The variance, indicative of data dispersion, is ascertained by computing the average squared difference between each payload length and the mean within the window, denoted in Eq. (4) as fma5_variance: (4)${\displaystyle \begin{array}{c}\hfill {\mathrm{fma5}}_{\mathrm{variance}}=\frac{1}{5}\sum _{j=1}^5{\left(P_{i+j}-{{\mathrm{fma5}}_{\mathrm{mean}}}_i\right)}^2.\hfill \end{array}}$

Then, the standard deviation, reflecting data scatter, is the square root of the computed variance, as shown in Eq. (5): (5)${\displaystyle \begin{array}{c}\hfill {\mathrm{fma5}}_{\mathrm{std}}=\sqrt{{{\mathrm{fma5}}_{\mathrm{variance}}}_i}.\hfill \end{array}}$

Payload length range for the initial five packets (range_5)

This feature gauges the range between the maximum and minimum payload lengths among the first five packets of a traffic flow: (6)${\displaystyle \begin{array}{c}\hfill {\mathrm{range}}_5=max\left(\sum _{i=1}^5{P}_i\right)-\min \left(\sum _{i=1}^5{P}_i\right).\hfill \end{array}}$

Maximum payload length in either traffic direction (max_payload)

The last feature evaluates the maximal payload length between two traffic directions: source to destination (max_a) and destination to source (max_b). The formulation is as follows: (7)${\displaystyle \begin{array}{c}\hfill {\max }_{\mathrm{payload}}=\mathrm{maximum}\left({\max }_a,{\max }_b\right).\hfill \end{array}}$

where max_a signifies the average payload length computed from the source to the destination within a given traffic flow, and max_b represents the average payload length calculated from the destination to the source for the same traffic flow.

After completing the feature preparation, the processed dataset includes the necessary labels for classification. To make the labels suitable for machine learning algorithms, they are transformed into an appropriate categorical representation using the Label Encoder provided by the Scikit-learn library. This transformation process is essential as it converts categorical features into a suitable numerical representation, ensuring the data is well-prepared for machine learning algorithms.

Furthermore, unlike other existing studies (Charyyev & Gunes, 2021; Liu et al., 2021a; Najari et al., 2020) that divide their dataset into training and testing data, this article follows the partitioning strategy as in (Kolcun et al., 2021) to evaluate accuracy degradation over time, which partitioned its datasets into several weeks intervals. However, we partitioned the datasets into one-week intervals due to the duration covered by the datasets utilized in our article—spanning six and eight weeks. The training set comprises data from the initial week (Week 1), while the testing set consists of data from subsequent weeks (Week 2 to Week 8). This strategic division of datasets allows for a comprehensive assessment of the model’s performance over an extended period, ensuring its efficacy in handling IoT device identification with improved accuracy.

Identification phase

This article introduces a hybrid identification technique that combines the following approaches as follows:

1. Random Forest algorithm. Widely adopted in IoT device identification models, Random Forest is a supervised machine learning method renowned for its high accuracy and efficiency (Hamad et al., 2019). It achieves its accuracy and effectiveness by aggregating predictions from multiple decision trees, effectively reducing the risk of overfitting.

2. OneVsRest strategy. This strategy constructs individual classifiers for each class and trains them against all other classes (Miettinen et al., 2017). Doing so simplifies the labelling process for new IoT devices and enables a detailed analysis of each class by examining its respective classifiers.

The hybrid approach leverages the strength of the Random Forest algorithm’s ensemble method and the versatility of the OneVsRest strategy to achieve accurate and efficient IoT device identification. The parameters listed in Table 4 play a crucial role in optimizing the Random Forest algorithm’s performance, ensuring its robustness and effectiveness in handling diverse IoT environments.

Evaluation phase

The evaluation phase involves evaluating the machine learning model’s performance using various metrics. This evaluation process yields valuable insights into the model’s effectiveness. The metrics employed in this article are as follows:

(a) Accuracy. The accuracy gauges the overall correctness of the model’s predictions. It quantifies the proportion of correctly classified instances out of the total instances in the dataset. The accuracy is calculated using the formula in Eq. (8): (8)${\displaystyle \begin{array}{c}\hfill \mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{TN}+\mathrm{FN}}.\hfill \end{array}}$

(b) F1 score. The F1 score offers a comprehensive evaluation, especially in imbalanced datasets, ensuring a more reliable assessment of the IoT device identification system’s performance. The F1 score is calculated using the formula in Eq. (9): (9)${\displaystyle \begin{array}{c}\hfill F1\mathrm{score}=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.\hfill \end{array}}$

The definitions for the components of these formulas are as follows:

• True positive (TP). The instances where the model correctly identifies device A as device A and device B as device B.

• True negative (TN). The instances where the model correctly identifies data that is not device A as not device A and data that is not device B as not device B.

• False positive (FP). Represents false alarms or incorrect positive predictions.

• False negative (FN). Represents missed positive predictions.

• Precision. Measures the proportion of actual positive instances the model correctly identifies.

• Recall. Measures the proportion of instances the model correctly identified as positive out of all predicted instances.

These metrics enable a comprehensive evaluation of the model’s performance, facilitating a robust analysis of the IoT device identification system’s capabilities.

Accuracy & F1 scores degradation over time

In this section, we present the accuracy and F1 scores degradation analysis for both the proposed feature set and the flow-based feature set on the IoT Traffic Traces and IoT-FSCIT datasets in Tables 5 and 6, respectively. The experimental results for each dataset are summarised below:

Accuracy scores on public dataset (IoT Traffic Traces):

1. The proposed feature set initially achieved an accuracy of 89.13%, gradually declining over eight weeks to its lowest point of 83.05%.

2. Throughout the two-month testing period, the proposed feature set consistently maintained an accuracy level above 80%, with an overall degradation of 6.8%.

3. In contrast, the flow-based feature set exhibited lower accuracy scores compared to the proposed feature set throughout the testing period.

4. The flow-based feature set consistently dropped below 80% accuracy on most weeks, indicating a less reliable performance.

5. Despite the lower degradation observed in the flow-based feature set, the proposed feature set outperformed it in terms of overall accuracy scores, maintaining above 80% accuracy on all weeks.

6. Moreover, the proposed feature set achieved higher accuracy while utilising less than half of the features proposed in a previous article, aligning with the objective of proposing a robust feature set against accuracy degradation over time.

Accuracy scores on our dataset (IoT-FCSIT):

1. Proposed feature set: Initial accuracy of 99.5%, final accuracy of 89.69%, and degradation of 10.81% over the evaluation period.

2. Flow-based feature set: Initial accuracy of 99.83%, lowest accuracy of 78.89% (Week 4), and degradation of 20.94% over the evaluation period.

3. The proposed feature set consistently outperformed the flow-based feature set in accurately classifying devices within the IoT-FSCIT dataset.

4. The results of the comparison show that the proposed feature set is superior in terms of accuracy degradation, demonstrating its robustness against accuracy deterioration in different network environments.

5. Additionally, the proposed feature set achieved higher accuracy while utilising fewer features than in a previous article, highlighting the efficiency and effectiveness of the selected features.

6. Both approaches achieved higher accuracy scores in the IoT-FSCIT dataset compared to the IoT Traffic Traces dataset, attributed to the reduced complexity and smaller number of IoT devices in the IoT-FSCIT dataset.

In conclusion, the accuracy degradation analysis on both datasets demonstrates the superior performance of the proposed feature set over the flow-based feature set. The proposed feature set showcases robustness against accuracy degradation, highlighting its potential for real-world applications in diverse IoT environments. The findings are further analysed in the next section.

F1 scores on public dataset (IoT Traffic Traces):

We notice distinct trends in the performance of both the proposed feature set and the flow-based feature set when evaluating the IoT Traffic Traces dataset over an eight-week testing period, as shown in Fig. 3, as follows:

1. During the first week of testing, both methods exhibited remarkable performance, with the proposed feature set reaching an F1 score peak of 88.90% compared to the flow-based feature set’s lower score of 82.77%.

2. Week 2 saw a slight dip in the F1 scores of both methods. The proposed feature set came in at 83.85%, and the flow-based feature set wasn’t far behind at 81.17%.

3. From Week 3 through Week 6, the proposed feature set displayed a competitive edge, consistently scoring higher F1 scores than the flow-based method.

4. The gap between the two methods widened significantly during Week 6, with the proposed feature set achieving an F1 score of 83.60%, overshadowing the flow-based feature set’s score of 78.37%.

5. In the final weeks, we see both models improving but maintaining the performance gap. This observation hints at the robustness of the proposed feature set.

6. The proposed feature set holds its lead with an average F1 score of 84.90%, while the flow-based feature set trails with an average of 80.48%.

F1 scores on our dataset (IoT-FCSIT):

Following are the trends of the proposed features set and flow-based feature set over a six-week testing period, as shown in Fig. 4:

1. In the initial week, both feature sets demonstrate impressive performance. The proposed feature set achieves a near-perfect F1 score of 99.5%, while the flow-based feature set surpasses it slightly with an F1 score of 99.83%.

2. The second week sees a minor performance drop in both feature sets. Despite this dip, the flow-based feature set continues to hold a slight edge.

3. From Week 3 onwards, a dramatic shift occurs. The proposed feature set consistently outperforms the flow-based feature set up until Week 6. The disparity between the two feature sets is most pronounced in Week 4.

4. Over the entire period, the proposed feature set outshines the flow-based feature set with an average F1 score of 96.09% compared to the flow-based feature set’s average of 88.17%.

5. This substantial difference of approximately 7.92% in favour of the proposed feature set is not trivial. It highlights the superior predictive capability and stability of the proposed feature set over time, even when faced with datasets extracted from different network environments.

In conclusion, the F1 score degradation analysis on both datasets reiterates the robustness and superiority of the proposed feature set over the flow-based feature set. The findings highlight the effectiveness and stability of the proposed features in accurately classifying IoT devices over time, contributing to enhanced IoT traffic analysis in various network environments. Additionally, evaluating the IoT-FSCIT dataset showcases the adaptability of the proposed features, yielding improved accuracy and performance in a reduced complexity setting. These results further underscore the importance of selecting and refining features to address accuracy degradation and concept drift challenges in machine learning applications.

Misclassification among streaming devices

We ran the confusion matrix to provide better visualisation of misclassifications in both approaches on both datasets. The confusion matrix is a performance evaluation tool in machine learning to summarise the predictions made by the classification model and gain insights into its performance. The confusion matrix analysis on both the IoT Traffic Traces and IoT-FSCIT datasets revealed notable misclassifications among streaming devices. Complex data transmission processes and diverse network behaviours characterise streaming devices.

On the IoT Traffic Traces dataset, we ran the confusion matrix on Week 8 of the flow-based features set, as depicted in Table 7. The findings are as follows:

1. Amazon Echo: Misclassified as Belkin Wemo Switch 2.25%, Belkin Wemo Motion Sensor 2.52%, and HP Printer 1.75%.

2. HP Printer: Misclassified as Amazon Echo 0.41%, as Belkin Wemo Switch 5.58%, Belkin Wemo Motion Sensor 0.75%, Insteon Camera 2.01%, Samsung SmartCam 0.76%, and as Triby Speaker 0.93%.

3. Insteon Camera: Misclassified as Belkin Wemo Switch 0.42%, Belkin Wemo Motion Sensor 3.33%, and Samsung SmartCam 0.20%.

4. Samsung SmartCam: Misclassified as Belkin Wemo Switch 0.01% and Insteon Camera 0.15%.

5. Triby Speaker: Misclassified as Belkin Wemo Switch 0.06% and Insteon Camera 0.10%.

Table 8, the confusion matrix on Week 6 for proposed features set on IoT-FSCIT dataset also reveals misclassifications among streaming devices, which include Alexa Echo Dot, Mi Box 3, and Mi Home Security Camera. The findings are as follows:

1. Alexa Echo Dot: Misclassified as Mi Box 3 3.6% and as Mi Home Security Camera 0.7%

2. Mi Box 3: Misclassified as Alexa Echo Dot 6.42% and as Mi Home Security Camera 2.55%.

3. Mi Home Security Camera: Misclassified as Alexa Echo Dot 0.51% and as Mi Box 3 0.43%.

The confusion matrix analysis on both the IoT Traffic Traces and IoT-FSCIT datasets revealed significant misclassifications among streaming devices, indicating challenges in accurately identifying such devices. On the IoT Traffic Traces dataset, the flow-based features set displayed misclassifications primarily among Amazon Echo, HP Printer, Insteon Camera, Samsung SmartCam, and Triby Speaker. These devices were frequently misclassified as other streaming devices, suggesting similar network traffic patterns. Similarly, the proposed feature set on the IoT-FSCIT dataset exhibited misclassifications among streaming devices, including Alexa Echo Dot, Mi Box 3, and Mi Home Security Camera.

Additionally, we can observe the accurate and high scores achieved for devices like LiFX Smart Bulb, NEST Protect Smoke Alarm, NETATMO Weather Stations, Smart Things, and Withings Smart Scale on the IoT Traffic Traces dataset and Mi Air Purifier on the IoT-FSCIT dataset. These findings emphasize the relative ease of correctly identifying non-streaming devices with straightforward network traffic patterns.

As the confusion matrix in Table 7 shows, HP Printer, categorised as one of the streaming devices, exhibited the lowest accuracy score of 35.1%. To investigate the underlying cause of this degradation, we conducted a Kolgomorov-Smirnov (KS) test on the ‘Bidirectional Mean of Packet Size’ and ‘Source to Destination Bytes’ features for HP Printer between Week 1 and subsequent weeks (Weeks 2 to 8) in the flow-based approach. The KS test results for both features are visualised in Figs. 5 and 6, respectively. The KS Test assesses the similarity between two distributions, with a small p-value indicating a significant difference and a larger p-value suggesting similarity, where the significance level is set to 0.05. The findings revealed the following:

1. Bidirectional Mean of Packet Size feature:

   1. Week 2 vs. Week 1: The p-value is 0.0529, relatively close to significance. The distributions are considered the same, as the p-value is insignificant

   2. Weeks 3 to 8 vs. Week 1: The p-values for Weeks 3 to 8 are extremely small, ranging from 1.347e−16 to 1.124e−78. This indicates the distributions of the ’Bidirectional Mean of Packet Size’ feature for HP Printer significantly differ from Week 1 in all subsequent weeks.

2. Source to Destination Bytes feature:

   1. Week 2 to Week 8 vs. Week 1: The p-values for all testing weeks, Weeks 2 to 8, are extremely small, ranging from 1.69e−4 to 2.11e−81. This indicates the distributions of the ‘Source to Destination’ feature for the HP Printer in all testings weeks differ from its training data, Week 1.

The confusion matrix in Table 8 shows that Smart Plug has big misclassifications, with only 64.3%. We again ran the KS Test on all seven features of the proposed features set. The results consistently show a p-value of 0.0 between Week 1 and subsequent weeks (Week 2 to 6), indicating that the distributions of these features for the Smart Plug are significantly different from Week 1. These significant differences in data distribution suggest that the network behaviour of the Smart Plug has undergone substantial changes over time, which explain the big misclassifications observed in Table 8. Figures 7 and 8 illustrate two of the seven proposed features.

In summary, the statistical changes observed over time in the feature further corroborate the accuracy degradation findings, as the differences in distributions become more pronounced as time progresses. The model performance’s degradation is observed as early as Week 2, as (Wan et al., 2023) suggests. This degradation highlights a common problem in machine learning: the performance of a model degrades when there are significant changes in the data distribution over time. Maintaining model accuracy and reliability becomes particularly challenging in IoT environments, where device behaviour and network conditions evolve dynamically. Addressing these statistical changes accordingly is the main objective of this article, in which our proposed feature set managed to maintain above 80% accuracy scores on all weeks on both datasets.

Misclassification among devices of the same brand

The confusion matrix also showcases the misclassification between devices from the same manufacturer or brand, such as the Belkin Wemo Switch and the Belkin Wemo Motion Sensor. This article found similar network signatures causing incorrect classifications.

1. We plot the pie charts showing the top five most used destination ports for both devices, further demonstrating the similarity in their network traffic patterns, as shown in Figs. 9 and 10.

2. The findings highlight the superior performance of the proposed approach, maintaining above 80% accuracy despite also getting caught up in confusion between Belkin devices.

Experimental analysis and technical findings summary

This section provides a comprehensive analysis of accuracy degradation on the IoT Traffic Traces and IoT-FSCIT datasets by evaluating the performance of the proposed feature set using various tools, including confusion matrix, feature value plotting, and the Kolgomorov-Smirnov test. The findings summarisation are as follows:

1. Confusion matrix analysis revealed significant misclassifications among streaming devices on both datasets, indicating difficulties in accurately identifying such devices.

2. Non-streaming devices like LiFX Smart Bulb and NEST Protect Smoke Alarm achieved high and accurate scores on the IoT Traffic Traces dataset, suggesting their straightforward network traffic patterns were easier to classify.

3. HP Printer, classified as a streaming device, exhibited the lowest accuracy score of 35.1% on the IoT Traffic Traces dataset, pointing to challenges in maintaining accuracy in dynamic IoT environments.

4. The Kolgomorov-Smirnov test showed substantial changes in data distribution over time for HP Printer’s features, further highlighting the impact of dynamic IoT conditions on model performance.

5. Despite misclassifications between devices from the same brand, such as Belkin Wemo Switch and Belkin Wemo Motion Sensor, the proposed feature set maintained an overall accuracy above 80%.

The technical analysis underscored the significant challenges in accurately classifying streaming devices, particularly due to their complex data transmission processes and diverse network behaviours. However, the proposed feature set demonstrated promising results, maintaining accuracy above 80% on both datasets. To enhance classification performance further, addressing statistical changes in data distribution over time and refining the model’s ability to distinguish between devices from the same brand are crucial areas of focus in dynamic IoT environments.