Mitigating adversarial attacks in federated learning based network traffic classification applications using secure hierarchical remote attestation and adaptive aggregation framework

Secure framework design

This study proposes the SHeRAA-FL framework to enhance the defensive measures against multiple adversarial attacks for FL-based NTC. The framework consists of three main mechanisms: remote attestation scoring, hierarchical training, and adaptive aggregation. Figure 3 shows a diagram of a high-level overview of SHeRAA-FL.

The proposed framework clusters the FL clients based on domain in a hierarchical topology to enhance data privacy. Each edge client belongs to a domain, and multiple domains $D_k$ can join the FL training. Each client within the domain belongs to the same organizational or administrative boundary. For example, a university campus network has two edge devices participating in FL, where each edge device serves as a gateway router for different faculty. Thus, both edge devices are clustered in the same domain. Besides that, the framework organizes the hosts in a hierarchical topology, which has global, domain, and edge tiers. The edge is the lowest tier, which contains FL edge clients such as firewalls, switches, or routers that filter and forward endpoint traffic. The client captures a sample of the endpoint packets and uses it as a local dataset ${\mathbb{X}}_{\mathrm{k}}$ for training. The clients have a TPM and data store module for storing attestation-related data. They also have a program containing instructions on training local models and participating in the attestation and FL process.

Meanwhile, the domain tier contains a local aggregator $A_k$ which verifies and aggregates the parameters { $A{P}_{kj},{\mathrm{W}}_{\mathrm{r}}^{\mathrm{k}}$} of FL clients within the domain $D_k$. The local aggregator has all the modules of normal clients with the addition of a domain verifier and an adaptive aggregation module. Besides that, local aggregators also perform the training tasks of normal clients. The local aggregator produces a domain-level model ${\tilde{f}}_{Domain}(.)$ by aggregating the weight parameters of clients within its domain. Lastly, the global tier contains a global aggregator server with TPM, a global verifier, and an aggregator module. The global aggregator produces a global model ${\tilde{f}}_{Global}(.)$ by aggregating the parameters of the domain model sent by the local aggregator of each domain.

Figure 4 outlines the flow of the FL training process within the SHeRAA-FL framework. The process begins with an initialization stage where the clients perform data preprocessing on their local datasets. Next, the global server and the clients generate public and private keys to create digital certificates. The framework requires these certificates to establish encrypted TLS communication and verify host identities. After that, the clients train an evaluation model and create a verification list, both of which are necessary for the remote attestation process. The server and clients then engage in a remote attestation scoring process. Then, the framework starts with the hierarchical training and adaptive aggregation process, which continues until the process reaches the maximum training round. Once training is complete, the server distributes the global NTC model to the clients for inference.

The following are details on remote attestation scoring, hierarchical training, and adaptive aggregation mechanisms.

• (a)

  Remote attestation scoring. Before FL training began, the framework’s first step was establishing trust among the FL server and clients via the remote attestation scoring process. Algorithm 1 shows the framework’s remote attestation scoring process. The proposed framework leverages hardware-level security, such as TPM, to store server and client FL program and verification parameters in a secure enclave to minimize the risk of tampering during the attestation and training process. This study selected the TPM for this study due to its wide availability in commodity computing hardware. In contrast, a TEE requires specialized processors that include features like Intel SGX or ARM TrustZone.

  The process starts with the initialization stage, where each client $k$ generate and prepares attestation parameters $A{P}_{ij}$ which are needed by the global server $GS$ to calculate client $k$ trust score $T_k$. The attestation parameters $A{P}_{ij}$ contains test model ${\tilde{f}}_{Test}(.)$ which are trained using client $k$ local dataset ${\mathbb{X}}_{\mathrm{k}}$, local dataset hash $HC{D}_k$, verification list $V{C}_k$, a hash of the client’s FL program $HC{P}_k$, attestation program $HAC{P}_k$, public certificate $HP{C}_k$, and verification list $HV{C}_k$. The training of the test model only involves several epochs to shorten the training time, as it is only used by $GS$ to evaluate the dataset quality. Meanwhile, for the $V{C}_k$, the clients’ attestation program provides the number of running processes $PS$ and open port $OP$ on the host. Besides that, the clients’ attestation program also checks ${\mathbb{X}}_{\mathrm{k}}$ from potential backdoor patterns. If the recurring pattern exceeds the threshold $Pt$ value, the attestation program sets the backdoor status $BS$ as True, remove the identified pattern and include the information in $V{C}_k$. The threshold value is hardcoded in the remote attestation program, and tampering with it causes the program’s hash value to differ from the global server reference value.

  After initialization, clients $k$ send the $A{P}_{ij}$ to $GS$ via TLS to ensure communication is kept private and the client communicates with the verifiable global server to avoid spoofing. When $GS$ receives the client $k$ $A{P}_{ij}$, first, it verifies the information by comparing it with the pre-defined client list $C=\left\{C_1,C_2,\dots ,C_k\right\}$. If there is a discrepancy with the client’s $I{D}_k$, domain number $D_k$, IP Address $I{P}_k$, public certificate $HP{C}_k$, or the attestation program hash $HAC{P}_k$ the server removes the suspected client data from training. Tampering with the $A{P}_{ij}$ indicates an attempt to bypass the attestation process or initiate a Sybil attack, which involves creating multiple fake identities (Xiao et al., 2022).

  Once verified, the valid clients’ hash values are stored in the server’s TPM to avoid tampering during the ongoing attestation process. The attestation scoring process is conducted per domain based on information in the attestation parameters $A{P}_{ij}$. First, client $k$ with verified FL client programs $HC{P}_k$ are given fifteen trust score $T_k$, while clients with tampered FL client programs are given zero points and added to untrusted client $U{C}_k$ list. Clients with tampered programs have a higher risk of being malicious, and attackers can manipulate the structure of the model and update parameters by tampering with the FL program. The proposed algorithm will take further action on clients with tampered datasets without resorting to simply discarding the potentially clean dataset. Second, clients with minimum $PS$ or $OP$ are given five points $T_k$ each, as clients with lower values have a lower risk of being compromised. Third, clients with no suspected backdoor pattern are given five trust points $T_k$.

  The fourth criterion is the clients’ test model ${\tilde{f}}_{Test}(.)$ performance. The client with the highest F1-score is given 10 points, while clients with an F1-score of more than 60% are given five points $T_k$. Higher test model performance is a good indicator that the client datasets have a lower risk of being poisoned, while models with more than 60% F1-score indicate the client has a reliable dataset for training the NTC model (Jenefa & Edward Naveen, 2023). The trust score assigned to each evaluation criterion is based on the severity of potential security risks. For example, tampering with the FL program has the highest severity; thus, a potentially malicious client is penalized fifteen points as tampering with the FL program enables attackers to modify the model structure, update parameters, and bypass security protocols. Meanwhile, clients with low test model accuracy are penalized with 10 to 5 points as it is a good indicator that the dataset has been poisoned. Moreover, the trust score value is also derived by experimentation during the design stage to ensure less risky clients are chosen as local aggregator $A_k$ and risky clients’ trust score $T_k$ are penalized to reduce their influence during the adaptive aggregation mechanism.

  After $GS$ calculates the trust score $T_k$ for each client $k$ in the domain, the client with the highest $T_k$ in the domain is selected as the local aggregator $A_k$ for the domain $D_k$. To uniquely identify the $A_k$, $GS$ generates an aggregator token $HA{T}_{Dk}$ and stores the token in TPM. The $GS$ sends the token to $A_k$ along with other information, such as trust score $T_k$, FL Client program hash $HCP$, local dataset hash $HC{D}_k$, $A{P}_{Dk}$, verification token $HA{T}_{Dk}$ and untrusted client list $U{C}_{Dk}$ of all clients $k$ in its domain $D_k$, which are needed during the hierarchical training process. Meanwhile, for other clients $k$, if the client is untrusted $U{C}_k$, $GS$ send verification token $H{T}_{Dk}$, local aggregator $I{P}_{A_D}$, local aggregator public certificates $P{C}_{A_D}$, along with a dataset upload request $UR$ which will be used for the training delegation process during hierarchical training. For other client $k$ not in untrusted client $U{C}_k$, $GS$ only send a verification token $H{T}_{Dk}$, local aggregator $I{P}_{A_D}$, and local aggregator public certificates $P{C}_{A_D}$. Both local aggregators $A_k$ and clients store the information they receive in TPM to prevent tampering.

• (b)

  Hierarchical training. After selecting local aggregator $A_k$ and assigning trust score $T_k$, the framework begins with the hierarchical FL training. Algorithm 2 shows the hierarchical training process of the framework. Client $k,$ which received aggregator token $HA{T}_{Dk}$ assume the role of $A_k$ and listens for client requests via TLS. Meanwhile, others become local clients and establish communication with local aggregators via TLS using local aggregator IP address $I{P}_{A_D}$ and public certificate $P{C}_{A_D}$ received from global server $GS$ earlier. If clients $k$ received dataset upload request $UR$ from $GS$ during remote attestation process, the client requires them to upload their local dataset ${\mathbb{X}}_{\mathrm{k}}$ to the selected $A_k$ of domain $D_k$ using its $I{D}_k$ and $H{T}_{I{D}_k}$ as identification.

  Meanwhile, clients $k$ which doesn’t receive $UR$ are only require to send the hash of their local dataset $HC{D}_k$ and their FL client program $CP$ to the selected $A_k$ of domain $D_k$ using its $I{D}_k$ and $H{T}_{I{D}_k}$ as identification. At this stage, only the suspected untrusted clients $U{C}_k$ are given a choice either to upload their local dataset ${\mathbb{X}}_{\mathrm{k}}$ to the selected $A_k$ of domain $D_k$ or being removed from the FL training. To ensure data privacy, the local dataset ${\mathbb{X}}_{\mathrm{k}}$ is uploaded only to $A_k$ in $D_k$, thus the local dataset ${\mathbb{X}}_{\mathrm{k}}$ is not exposed to the host such as $GS$, $A_k$ or client $k$ outside the organizational boundary. Moreover, to avoid unnecessary usage of bandwidth, only untrusted clients $U{C}_k$ requires uploading local dataset ${\mathbb{X}}_{\mathrm{k}}$ to $A_k$ which nearest to $U{C}_k$. This is to avoid simply discarding a good dataset although the client’s FL program has been compromised.

  After that, the selected $A_k$ perform verification process for clients $k$ in its domain $D_k$. The verification process is crucial to ensure the clients $k$ do not alter any data after the remote attestation process with $GS$. The initial steps involve the $A_k$ verifying the FL program $CP$ and local dataset hash $HC{D}_k$. The verification is done differently for untrusted and normal clients. If the client $k$ in untrusted client $U{C}_k$ list, first $A_k$ check the verification token $H{T}_{I{D}_k}$ and the uploaded ${\mathbb{X}}_{\mathrm{k}}$. If the token mismatch from information received from $GS$ or clients $k$ refuse to upload the ${\mathbb{X}}_{\mathrm{k}}$, $A_k$ removes the client from FL training along with its data. On the other hand, if the client $k$ upload ${\mathbb{X}}_{\mathrm{k}}$ to $A_k$ secure storage, $A_k$ delegates the client’s training task to other trusted clients.

  The delegation process involves generating a delegation token $HD{T}_{U{C}_k}$ and new $I{D}_{U{C}_k}$, then sending the untrusted client’s dataset ${\mathbb{X}}_{{UC}_{\mathrm{k}}}$ to the selected trusted client. The new $I{D}_{U{C}_k}$ instance will inherit the trust score $T_k$ of the untrusted client $U{C}_k$. If there is no suitable, trusted client to delegate, the training task will be delegated to the local aggregator $A_k$. This will ensure the $U{C}_k$ local dataset ${\mathbb{X}}_{\mathrm{k}}$ is not simply discarded and trained using a verified FL training program. Although there is a risk to the dataset ${\mathbb{X}}_{\mathrm{k}}$ is poisoned by the malicious client, the risk will be minimized by an adaptive aggregation mechanism. Meanwhile for normal clients $k$, $A_k$ check the verification token $H{T}_{I{D}_k}$, local dataset hash $HC{D}_k$ and FL client program hash $HCP$. If any of the values mismatch with information received from $GS$, $A_k$ Remove the client from FL training and its data as it indicates the client has malicious intentions. Before completing the verification process, $A_k$ once again updates its TPM to include changes related to token and trust score.

  Once the domain-level verification is done, FL training starts when the local aggregator $A_k$ sends the clients in domain $D_k$ with the initial NTC model weight $W_{r=0}$ and training hyperparameters. This is to ensure all local clients are training using the same model structure and parameters, such as $\mathrm{r}$ training round, $E$ epochs, $\eta$ learning rate, and $\beta$ batch size. The clients train the NTC model using its local dataset and then send its weight parameters along with a verification token. If the token is valid, the local aggregator aggregates the local model’s weight parameters via the adaptive aggregation mechanism. At the end of the training round, the local aggregator produces a domain model ${\tilde{f}}_{Domain}(.)$ based on parameter updates from the local clients. The adaptive aggregation process will be discussed in the following subsection.

  After training the domain model, the local aggregator in each domain sends the weight of the domain model to the global aggregator server along with the aggregator token $HA{T}_{Dk}$ for verification. If the token is valid, the global server aggregates the domain model’s weights using averaging methods and distributes back the weighted average to all local aggregators. After that, the local aggregator updates the weight of the domain model to form a global NTC model ${\tilde{f}}_{Global}(.)$ which is then distributed to all clients for inferences and classifying traffic.

• (c)

  Adaptive aggregation. During hierarchical training, the framework aggregates the weight parameters of the clients via an adaptive aggregation process, as detailed in Algorithm 3. It starts with checking the parameters update for a potential GAN attack. During the initial training round, the local aggregator $A_k$ forms an evaluation model ${\tilde{f}}_{Eval}(.)$ for each client $k$ including the local aggregator from the parameter update. The F1-score for each class in the local aggregator evaluation model becomes a benchmark for evaluating the possibility of a GAN attack in the clients’ parameter update. When evaluating the clients’ update, if it causes the number of classes with F1-scores less than 0.1 or 10% to exceed the threshold, it indicates that the client is attempting a GAN attack. In this study, the GAN attack threshold is set to two classes.

  After checking for GAN attack, the local aggregator calculates the clients’ weightage $C{W}_k$ based on three scenarios. First, if there is an attempt to send a poisonous GAN attack via an update, 10% of the weightage is divided among the suspected clients, while 90% is divided among the trusted clients. This limits the influences of the GAN attack while ensuring a portion of information from the malicious clients is learned. The second scenario is when there is no suspected GAN attack and the trust score ${\tau }_k$ of all clients equal to or more than the median value. This indicates an acceptable trust level among all clients and thus the weightage is divided equally among the clients so that each update has equal influence in the domain model ${\tilde{f}}_{Domain}(.)$. The third scenario is when there is no suspected GAN attack; however, some of the clients’ trust scores are below the median level. Thus, 20% of the weightage is divided among the risky clients, while 80% of the weightage is divided among the trusted clients to limit the influence of the risky clients.

  After calculating and assigning weight to each client, the local aggregator begins aggregating the clients’ parameter updates. For the first round, aggregation is done via the standard FedAvg algorithm because, in the first round, the local model has not incorporated the weight ${\mathrm{W}}_{\mathrm{r}+1}^{\mathrm{k}}$ learned from other clients within the domain $D_k$. However, in the second round, the local aggregator aggregates the clients’ parameters using multiple algorithms to form multiple test model $\tilde{f}Tes{t}_i(.)$. The algorithm includes FedAvg, Weighted averaging, Krum, Multi-Krum, Median, and Trim Mean. The local aggregator evaluates the performance of each test model and algorithm with the highest accuracy is selected as the best aggregation algorithm $AG{G}_{Best}()$. This is because some algorithms are more effective in certain conditions or attacks. For example, FedAvg provides the best performance in normal conditions, while a distance-based approach such as Krum is more effective for LF attacks.

  Moreover, during aggregation, the local aggregator also set the weight for each client’s parameters based on the assigned weight $C{W}_k$ calculated earlier to increase or decrease the clients’ influences. For the following training round, the local aggregator uses the selected best algorithms for aggregation. At the end of each training round, the local aggregator returns the updated weight ${\mathrm{W}}_{\mathrm{r}+1}^{\mathrm{k}}$ back to the clients.

Federated learning testbed setup

The FL testbed consists of a single aggregation server and six edge clients; each of the hosts was assigned a different TCP/IP port number. Each of the edge clients trains a local model using its local dataset ${\mathbb{X}}_{\mathrm{k}}$ and sends the model parameters ${\mathrm{W}}_{\mathrm{r}}^{\mathrm{k}}$ to the aggregation server to form a global model ${\tilde{f}}_{Global}(.)$. The client trains a DL-based NTC model, which involves processing the raw network packet byes into labeled packet byte matrix (PBM) (Wang et al., 2018). This study uses MLP with two fully connected hidden layers, each with six nodes, and the rectified linear unit (ReLU) activation function. The input layers accept input with 740 packet bytes as features and use ReLU as the activation function. Meanwhile, the output layer classifies traffic into ten classes and uses Softmax as the activation function.

Besides MLP, we also used 1D-CNN and 2D-CNN architectures to train the local models. This ensures a fair benchmark against existing work by Sameera et al. (2024) and Thein, Shiraishi & Morii (2024). This study CNN architecture begins with a convolutional layer containing 64 filters, followed by a max-pooling layer, and then a second convolutional layer with 128 filters. Both convolutional layers use a kernel size of 3 and the ReLU activation function. After the convolutions, we apply a flatten layer before a fully connected layer of 128 nodes, which also uses ReLU activation. To prevent overfitting, we include a dropout layer with a rate of 0.5 just before the final output. Since CNNs require input data in a specific pixel-like format, we reshaped the datasets accordingly. For example, we reshaped the Fashion-MNIST dataset into 28 × 28 × 1 arrays and the CIFAR-10 dataset into 32 × 32 × 3 arrays.

The model uses categorical cross-entropy as the loss function and ADAM as the training optimizer. The training learning rate is set to $\eta =0.001$, batch size is $\beta$ = 64, with FL training round $\mathrm{r}$ = 3 and each round epoch $E=36.$ During the experiment, several aggregation algorithms were used as a benchmark for the defensive effectiveness of the proposed framework. The FedAvg is the default FL algorithm and does not have any defense against adversarial attacks. Meanwhile, other robust aggregation algorithms include weighted averaging (WA), median mean (MM), trimmed mean (TM) with a 10% trim rate, Krum, and Multi-Krum.

For the implementation of the framework and experiment, this study uses Python 3.8, including libraries such as scikit-learn 1.5.1, PyShark 0.3.6, TensorFlow 2.12.1, CUDNN 8.9, Twisted 18.9.0, Flower 1.6.0 (Beutel et al., 2020) and WandB. The source code is made available on the GitHub (Ariffin, 2025). The experiment was conducted on a host with an AMD Ryzen 7 7840HS 8-core CPU, 16 GB DDR5 Memory, and an Nvidia RTX 4070 GPU. The host runs Ubuntu 20.04 LTS for its OS and other programs such as CUDA 12.6, IBM TPM 2.0 simulator, and TPM 2.0 libraries such as TSS 3.1.0, ABRMD 2.3.1, TSS-engine 1.1.0, Tools 4.3.2, and OpenSSL 1.1.2.

Datasets and preprocessing

This study uses the ISCX-VPN 2016 network traffic dataset (Draper-Gil et al., 2016) which contains packet capture of popular network protocols or services, including encrypted SSL/TLS traffic. This study selected ten traffic classes, which are: (0) AIM Chat, (1) Email, (2) Facebook Audio, (3) Facebook Chat, (4) Gmail Chat, (5) Hangouts Chat, (6) ICQ Chat, (7) Netflix, (8) Spotify, (9) YouTube. The total number of traffic packet instances selected is 671,326, and after randomly splitting training and evaluation datasets at a ratio of 70:30, the training instance becomes 469,928, and the evaluation instance becomes 201,398. Table 4 provides details about the ISCX-VPN 2016 dataset, including a breakdown of class instances and descriptions of the traffic data. This breakdown indicates an imbalanced distribution of traffic classes, a characteristic that researchers often find in network traffic data (Abdelkhalek & Mashaly, 2023). This study maintains this imbalanced distribution in the training datasets to simulate non-IID (non-independently and identically distributed) conditions when evaluating the FL solutions (Zhang et al., 2021b). Researchers often refer to this specific type of statistical heterogeneity as label distribution skew or class imbalance. It is a primary attribute of non-IID data, especially in FL environments where clients collect data from different network environments (Jimenez G et al., 2024).

The raw packet data in the ISCX-VPN 2016 dataset must undergo preprocessing before it can be used for FL training. The first step of the preprocessing is parsing the packet data to remove the data link layer or Ethernet header, as it only contains significant local information. Moreover, we also remove the source and destination IP address as the information is constantly changing and only significant for specific network environments. After that, the raw packet bytes undergo a padding or truncating process to limit or make the size of each instance within 740 bytes or half the size of the maximum transmission unit (MTU).

Then the raw bytes are transformed into packet byte vectors (PBV) $X_i=\left\{x_{i1},x_{i2}\dots x_{ij}\right\}$, where $i$ represents the dataset and $j$ represents the j-th byte in $X_i$. Each PBV needs to be associated with a traffic label $y$ (e.g., Email, Facebook, YouTube), with each byte in the PBV serving as an input feature. After that, we aggregate all PBV together with its labels to the form of $\mathbb{X}=\{X_1^T,X_2^T\dots X_i^T{\}}^T$, where $i$ is the number of PBV datasets. Then we combine all processed PBV into a packet bytes matrix (PBM), where each row represents packet instances, and each column represents byte features along with its label; as such $\mathbb{X}=\left[\begin{array}{c}{X}_1\\ ...\\ X_i\end{array}\right]\leftarrow$ $\left[\begin{array}{c}{\mathrm{y}}_1\\ ...\\ {\mathrm{y}}_i\end{array}\right]$, where ${\mathrm{y}}_i$ is the traffic label. The label is then converted into a one-hot encoding format. After converting to PBM format, we normalize the values via the L2 normalization method along the y-axis as the DL model achieved faster convergence and higher performance with normalized value. The last steps of pre-processing involve dividing both training and evaluation datasets into six data shards, ${\mathbb{X}}_1,{\mathbb{X}}_2\dots {\mathbb{X}}_k$ which are used by the edge clients as their local datasets. Each of the clients’ data shards maintains a skewed distribution of traffic classes to create non-IID characteristics. In addition to ISCX-VPN 2016, this study uses other public datasets, such as N-BaIoT, Fashion-MNIST, and CIFAR-10. The N-BaIoT dataset (Meidan et al., 2018) contains real network traffic data that researchers collected from multiple commercial IoT devices infected with popular botnet malware. Table 5 provides details regarding the data instances of the N-BaIoT datasets. This study uses the N-BaIoT dataset to provide a fair comparison with the reported value in the work of Thein, Shiraishi & Morii (2024); therefore, we preprocessed the data following the methodology they described in their work. Meanwhile, the Fashion-MNIST dataset (Xiao, Rasul & Vollgraf, 2017) contains 70,000 instances of grayscale (single-channel) images, and each image is a 28 × 28 matrix of pixel intensities. The dataset has 10 classes that represent different types of clothing and fashion accessories, such as T-shirts/tops, Trousers, and Sneakers. This study uses this dataset to provide a fair comparison with the reported value in the work of Sameera et al. (2024).

Lastly, the CIFAR-10 dataset (Krizhevsky, 2009) contains 60,000 instances of RGB (3-channel) color images. Each image is a 32 × 32 matrix with pixel values that range from 0 to 255. The dataset has 10 classes that represent common objects from various categories, including animals, vehicles, and transportation. This study uses this dataset to provide a fair comparison with reported value in the work of Cao et al. (2024). For our experiments, we divide the N-BaIoT, Fashion-MNIST, and CIFAR-10 datasets into six training and evaluation data shards for the clients.

Attack scenarios

During the evaluation, this study simulates different adversarial attacks while training the NTC model via FL. This study conducts multiple experiments for each attack type. The experiment starts with a normal scenario with no attacker, which serves as a control or baseline. Then, we repeat the experiment with one, two, and up to four attackers to simulate a collusive environment. This study simulates the following four types of adversarial attacks:

• (a)

  Label flipping attack. This study simulates two variants, which are all and class label flipping (LF) attacks. Both attacks involve tampering with the local datasets of participating FL clients. The all-LF involves flipping the label of all classes. Where ${\mathbb{X}}_i$ has a true label $y_i$ for class $C_i,y_i\in \{C_1,C_2,\dots ,C_n$ and flipping function as $f:y_i\to y_i^{\prime }$. The attack aims to degrade overall classifier performance. Meanwhile, the class-LF involves flipping only the label of FB-audio (class no 2) traffic to degrade the class performance. Where in ${\mathbb{X}}_i$ if it’s $C_{FBAudio}$ flips to a random label $C_i$, otherwise, the label remains unchanged $f\left(y_i\right)=\{\begin{array}{rl}& C_{i}if{y}_i=C_{FBAudio}\\ & y_{i}otherwise\end{array}$.

• (b)

  Model poisoning attack. The attacks involve tampering with clients’ FL programs to manipulate the model structure and the values in the parameters update. This study simulates two variants of the attack: model cancelling and gradient factor attack. To conduct the model cancelling attack, the malicious client sets the model’s weight to zero, $\mathrm{\Delta }{W}_m=0$ (Rey et al., 2022). Meanwhile, for the gradient factor attack, we multiply the client gradient with a negative factor $\alpha$, where $\alpha <0$ (Blanchard et al., 2017). Both attacks severely degrade the overall performance of the global model.

• (c)

  Backdoor attack. The attack is based on the work of Bagdasaryan et al. (2018) which involves injecting backdoor patterns during FL training with the aim of poisoning and causing the global model to misclassify network traffic during inference without affecting the overall accuracy; thus, it is a more targeted attack. For example, when using the ISCX-VPN 2016 datasets, Email traffic (Class No. 1) is selected as the target class $Y_{Target}=1$, and FB audio (Class No. 2) as the malicious class $M_{Malicious}$ = ${\mathbb{X}}_{\mathrm{k}}\left[:,2\right]$. Then, this study sets the poison sample number $m$ = 20,000 which is used by malicious clients. After that, the backdoor script creates a backdoor pattern $X_B$ for 740 features based on $m$ values, values = $c/m$ where $c$ is the current feature iteration, which starts at zero and increments until it reaches 740.

  The script will produce a pattern $X_B$ with 740 columns. Then, during training, the malicious clients randomly select $m$ data instances from their dataset shard and replace the values of the selected instances with the generated backdoor pattern. The malicious clients also assign the backdoor data instances’ label as target class label, $Y_{\mathrm{k}}$[ $Y_{Random},Y_{Target}$] = 1. During the evaluation, this study injects the same backdoor pattern $X_B$ into the FB Audio instance to make the model misclassify the traffic as Email traffic. The attack poison rate is calculated as $P_{rate}={\displaystyle \frac{m}{Total({\mathbb{X}}_{\mathrm{k}})}\times 100}$, where $Total({\mathbb{X}}_{\mathrm{k}})$ is the total number of instances in the training dataset. Therefore, in a scenario with one attacker, the rate is calculated as $P_{rate}={\displaystyle \frac{\mathrm{20,000}}{\mathrm{469,928}}\times 100=4.25\mathrm{\%}}$. The poisoning rate doubles as the number of attackers increases, since each attacker uses the same poison sample value of $m=\mathrm{20,000}$.

• (d)

  GAN-based attack. This targeted attack involves generating synthetic data using a generative adversarial network (GAN) model for specific traffic classes and injecting it during FL training to cause a bias during classification (Zhang et al., 2019). Before conducting a GAN attack, it is necessary to train both the generator and discriminator models. The generator’s input layer takes random noise with a shape of 200, while the output layer synthesizes 740 features mirroring real traffic data and uses the sigmoid activation function. Meanwhile, the discriminator’s input layer accepts synthetic data from the generator and real data samples with a shape set to 740. The discriminator model aims to distinguish between real and synthetic data and then calculate the losses for both the generator and discriminator, guiding the improvement of their gradients during training. In this study, we use the trained generator model to synthesize FB Audio traffic and inject the synthetic data into Netflix as target class during training to make the global model bias to classify Netflix (Class no 7) as FB Audio (Class no 2) traffic. The malicious edge client replaces the dataset ${\mathbb{X}}_{\mathrm{k}}$ target class data in local dataset with the generated synthetic data, ${\mathbb{X}}_{\mathrm{k}}$[ $Y_{Target}$] = $x_{Synth}$. Similar to backdoor attacks, for the GAN-based attack, each malicious client sets the poison sample number $m=\mathrm{20,000}$. The GAN poison rate increases as the number of attackers increases, $P_{rate}={\displaystyle \frac{m}{Total({\mathbb{X}}_{\mathrm{k}})}\times 100}$, where $Total({\mathbb{X}}_{\mathrm{k}})$ represents the total number of training datasets from all malicious clients.

Evaluation metrics

This study uses overall accuracy, F1-score, and attack success rate (ASR) metrics to evaluate the performance of the NTC model and the effectiveness of the SHeRAA-FL framework in mitigating adversarial attacks. Both accuracy and F1-score metrics provide statistical validation for evaluating model performance in classifying multi-class network traffic, including imbalanced data scenarios. For LF, model cancelling, gradient factor, backdoor, and GAN-based attacks, we used the overall accuracy metric to measure the attack’s impact on overall model performance. We evaluate the defensive approach’s effectiveness by measuring its ability to maintain overall accuracy even when it receives poisonous updates from multiple attackers. Flipped labels, modified gradients, or backdoor patterns cause the client update to become poisonous by misleading its weight.

Meanwhile, for class-LF and GAN-based attacks, we also used the F1-score to assess the attack’s impact on specific traffic classes, as that is the attack’s aim. An effective defensive approach must prevent a reduction in a specific class’s F1-score that results from misleading labels or synthetic data injection. Previous studies have used both overall accuracy and F1-score metrics as common benchmarks to evaluate the effectiveness of defensive measures (Cao et al., 2024; Sameera et al., 2024; Rey et al., 2022; Thein, Shiraishi & Morii, 2024; Zhang et al., 2019).

Lastly, for the backdoor attack evaluation, this study uses the ASR in addition to overall accuracy. We calculate the ASR based on the number of successful backdoor attacks out of five attempts during inference. We record a backdoor attack as successful when the attacker misclassifies a target class (e.g., FB Audio to email) by injecting the backdoor pattern during inference. In each attempt, the model completes three FL training rounds $\mathrm{r}$ = 3. An effective defensive approach must prevent the FL model from misclassifying a class when an attacker injects a backdoor pattern during FL training. A previous study also used the attack success rate metric to evaluate defenses against backdoor attacks (Cao et al., 2024). This study also measures the FL training time, maximum CPU, and memory utilization as metrics for evaluating the proposed framework’s computational overhead.

Label flipping experiment

For the LF attack experiment, this study conducted two variations: all-LF and class-LF attacks. Table 6 presents the overall global model accuracy for the all-LF experiment using the ISCX-VPN 2016, Fashion-MNIST, and N-BaIoT datasets. Table 7 presents the F1-score of the target class (FB Audio) for the class-LF experiment using the ISCX-VPN 2016 dataset.