FLMatchQA: a recursive neural network-based question answering with customized federated learning model

Common information features

A single participant’s labeled data typically needs to be revised to train a reliable quality assurance model. In the FLMatch structure, to gather the shareable QA matching expertise amongst many participants, a shared backbone is proposed to mitigate the data sparsity issue. This model uses a cutting-edge neural matching model to determine if a potential response to a question is relevant.

Domain knowledge maintenance

Sharing the same model amongst participants may not be the best course of action because the QA data held by each participant may differ in size and properties. To mitigate the statistical heterogeneity, this article utilizes a private patch tailored to the unique domain information of each participant. Each participant’s backbone is patched, and the patch is trained only using the relevant private local QA data. As a result, the features particular to each participant are evaluated by the patch component and help to create a distinctive model for them.

Privacy-based learning

By sharing all model parameters or training samples, participants may compensate for a lack of data, even at the expense of their privacy. Thus, this model suggests employing FL (Shahamiri, 2021) technology to maximize FLMatch’s performance. This work merely uploads the local shared module’s parameters—typically containing less sensitive data about privacy—to the central server.

Adapter architecture

Determining the structure of the UMHFLM modules that will be developed is the first step in this process. Although several modules have been presented, this study concentrates on the Adapter because of its versatility in various domains, including audio and vision-and-image, and its proven effectiveness in completing specific duties. Within each block of the UMHFLM, the adaptor comprises feed-forward and down-projection functions interspersed between the feed-forward and self-attention layers. The process of adapting can be expressed as shown in Eq. (7) (7)${\displaystyle Adptio{n}^l\left(x\right)=U^{l}GeLU\left(d_x^l\right)+x}$

where, $d_x^l$ represents the weights for the down- and up-projection in the UMHFLM’s l^thlayer, respectively; d denotes the UMHFLM’s hidden dimension and ‘b’ the bottleneck dimension.

User embeddings’ construction

This work considered two forms of information to represent the clients’ characteristics: (a) label embeddings and (b) context embeddings (Duan et al., 2021a; Duan et al., 2021b). The explicit information on class distribution for each client is communicated in part through label embedding. The label distributions on mini-batches can adequately represent the data distributions of clients because they are often sampled using a uniform distribution. As a result, this model builds label embeddings using the mini batches’ label distributions. The label embeddings can be obtained in Eq. (8) if Batch = Di represents the client i’s mini-batches: (8)${\displaystyle Length\left(Batch\right)=W_{Length}\left(x_l\dots ..x\left|Batch\right|\right)+Batc{h}_{Length}}$ where avg(.) indicates average pooling within mini-batches, W_length ∈RC ×x and Batch_Length ∈R_x are the linear transformation weights and biases for the number of classes C, and ‘x’ is the dimensionality of input embeddings. Additionally, x_i is a one-hot label vector for the instance x_i,[;]. These values are represented by the concatenating function and batch length, respectively. Notably, this work chooses a uniform distribution for the inference phase to produce adapters that are not biased toward dominating classes because the test data labels are unavailable. By adopting a more comprehensive perspective, considering the contextual information in the data can help improve this work understanding of each customer (e.g., languages, text styles). To be more precise, layer-specific adapters are created by extracting contextual information from each layer. By averaging word vectors over the lengths using l₂ normalization, context embeddings are extracted, drawing inspiration from the sentence embeddings. Assuming that the sample vectors x_j from the PLM’s l^th layer are f_l(x _j), the l^th layer’s context embeddings can be obtained in Eq. (9): (9)${\displaystyle f^l\left(Batch\right)=W_{f}Max\left(f^l{x}_1,\dots ,f^l{x}_{Batch}\right)+Batc{h}_f}$

where W_f R_d ×x and Batch_f = ∈ R_x are the linear transformation weights and biases, respectively, and Max(.) indicates the max-pooling over the batch. Two types of embeddings are added together to provide comprehensive client embeddings. To further encourage the generator to encode more diversified layer-wise information, the study additionally appends layer-index embeddings into the client embeddings of each layer.

Client-conditional hyper networks

This research customizes the adapters for each heterogeneous client based on the client embeddings (Sattler et al., 2020). This work describes the client-conditional hypernetworks, which produce adapter parameters by taking the client embeddings lab as inputs. They are inspired by the idea of hypernetworks that generate parameters based on supplied input embeddings. Formally, Eq. (10) shows how hypernetworks act to create the adapter parameters (i.e., U_l, D_l): (10)${\displaystyle U_{Batch}^l{D}_{Batch}^{l}h\left(I_B\right)=\left(W_U,W_D\right)I_{Batch}^{Length}}$

where the weights for the hyper networks are represented by the letter ‘I’, which stands for the input embedding x, $W_D^{length}$ = $R^{r\ast d\ast s}{W}_U^{length}$ = R^d∗r∗N. It should be noted that various layers exchange hypernetworks with encoded layer-specific data for input embedding.

Factorization of hyper networks

Although hypernetworks can be used to create customized adapters, hypernetworks usually include many parameters. Thus, the proposed hypernetworks are factorized into two smaller weights (Fotouhi et al., 2024). The resulting parameters are also not skewed towards any of the local majority classes in the data distribution of the client because the resultant matrices from the factorized components are l₂ normalized. Two factorized parts are used to recreate the up-projection weights in Eq. (11) formally: (11)${\displaystyle U_{Batch}^l=WU{I}_{BBatch}=\partial \left(f_U{s}_U\right)I_{Batch}}$

where x, α(.) indicates the normalization and f_u = R^d∗s, S_u = R^d∗s∗r shows the factorized components from WU with latent factor x.

Regarding factorization, the expressivity and complexity of the resultant adapters are determined mainly by the hidden factor ‘s.’ To account for the increased dimensionality of latent components, the two projection weights are coupled in the same way as if they were tied auto-encoders (i.e.,) $D_{Batch}^{Length}$ = $U_{Batch}^{Length}$. This method allows the memory needs to be met without sacrificing task precision.

Aggregation phase

The corresponding trained models are sent back to the centralized server to update the global model when the training step on each client’s data is complete. Every client sends the layer-index embeddings and the hypernetwork parameters to the server, updating the global hypernetworks since the training models are hypernetworks.

Template-based QA (TQA)

The focus of quality assurance (QA) techniques has historically relied on extracting responses from raw text. Ontologies are used to annotate Web resources and enhance retrieval using query expansion (Enesi et al., 2019). Because of this (Duan et al., 2021a; Duan et al., 2021b), TQA was selected to train a recursive neural network model for template categorization. To obtain the answer(s), a SPARQL query template is essentially the last query’s blueprint, which must be created from the question. Query building, relationship extraction, and named entity recognition and disambiguation are most QA systems’ primary quality assurance activities. There will only be a solution that works flawlessly in some situations or domains (Bao et al., 2019). As a result, specific environments for which the QA components are experts have been developed. These components may be bootstrapped into modular question-answering pipelines. Most quality assurance systems do the conversion of queries into triples and compare them to a current knowledge base using similarity or ranking criteria to obtain the answer. Nevertheless, these triples frequently need to capture the natural language question’s semantic structure, leading to incorrect answers or poor SPARQL queries. On the other hand, this work’s domain-independent method can be used in any domain with minor modifications to the neural network. The necessary representations are automatically learned by RNN employing labeled instances furnished in the TQA dataset (Casado et al., 2021). The dataset uses a list of seed entities and predicates, enabling list filtering to form sub graphs of DBpedia for instantiating SPARQL templates, which in turn provide acceptable SPARQL searches. These SPARQL queries are subsequently used to instantiate Normalized Natural Question Templates (NNQTs) (d’Hondt et al., 2019; D’hondt et al., 2020), which serve as canonical structures but are often grammatically incorrect. These questions are manually edited and paraphrased by researchers.

Communication costs

Following McMahan et al. (2016), this article preserved the communication round number at which a target model accuracy of 92.8% had been attained in every experiment. By using the accuracy of the target model as a benchmark, this study assessed the communication costs across many studies. For each experiment, this work counted the number of models sent in the network at the round, where a 92.8% model accuracy was obtained to determine the communication cost. The number of models provided by the central server and each client was contained in FLAvg and was calculated in Eq. (13):

(13)${\displaystyle CentralizedFL=N\ast R\ast X\ast 2+x}$

where X is the total number of clients in the network, × is the percentage of clients the central server communicates with, and N denotes the round.

When 97% average model accuracy was obtained in the FLAvgP2P experiments, the world tallied all the models submitted by each client, which was computed as follows Eq. (14): (14)${\displaystyle FLE2E=R\ast N\ast A\ast X.}$

In this case, N is round, and 92.8% is the percentage of neighbors a client communicates with. Since it is a complete graph, the number of neighbors a client has, represented by the symbol A (equal to 99 in all experiments), is the same for all. Lastly, X represents the total number of network clients.

Accuracy

The correct answers the FL system gives for a percentage of questions are called Accuracy. There is only one correct response for every question. The span prediction task’s accuracy is similar to EM and may be calculated using Eq. (15) as follows: (15)${\displaystyle Accuracy=EM=\frac{NoofCorrectAnswer}{NoofQuestions}.}$

Recall is the proportion of tokens in a correct answer that have been successfully predicted in a question, whereas precision indicates the percentage of token overlap between the correct answer and the predicted answer.

The TP indicate

s the tokens that are identical between the predicted and correct answers, the tokens that are not in the correct answer but are in the expected response are marked by the FP, and the tokens that are not in the expected answer but are in the correct answer are shown by the FN. Using Eqs. (16) and (17), the precision and recall can be calculated as follows: (16)${\displaystyle Precision=\frac{NoofTP}{NoofTP+NoofFP}}$ (17)${\displaystyle Recall=\frac{NoofTP}{NoofTP+NoofFP}.}$

An indicator of the accuracy of a test is the F1 score. It is the precision and recall weighted average. It provides the calculation for this score. It is calculated in tets instance by comparing each word in the prediction to each in the true answer. The F1-score is based on how many words the prediction and the truth share, Eq. (18)

(18)${\displaystyle F1=2\ast \frac{Precision+Recall}{Precision+Recall}.}$

The 15,000-question pre-processed dataset was divided into train and test datasets, with 80% of the data being training and 20% being test. There were 13,936 questions in the training dataset and 984 questions in the test dataset. On the test SQuAD and QALD-7 datasets, this RNN model’s obtained template classification accuracy was 92.8% and 61.8%, respectively. Equation (19) is used to calculate the accuracy: (19)${\displaystyle Acc\left(x,\overline{x}\right)=\frac{1}{N}\sum _{i=1}^{N}1\left({\overline{x}}_i=x_i\right)}$

where, for example, is the predicted value of the ith and “x” is the matching reality value. There is a total of N samples. Table 1 tabulates the hyperparameters of the model. The input vector was the 444-dimensional word vector concatenated. The optimizer used was the Adam Optimizer, with a small batch size of 25 instances. The loss function employed was cross-entropy loss, which has been shown to work better for tasks involving multivariate classification. Due to the limited training instances and periodic learning rate curtailment, the model needed strict regularization to prevent overfitting and improve its generalization performance. To do this, three strategies were employed:

1. Weight decay: The weights are modified by a part of the weight update rule known as Weight Decay, or l ₂ regularization, following each pass by calculating the product of a factor smaller than 1. This may be considered gradient descent on a quadratic regularization term (comparable to l₂ Normalization) and keeps the weights from getting too big. 2.25 ×10⁻³ was the weight decay utilized in the model.

2. Dropout: Powerful machine learning systems are deep neural networks that contain numerous parameters. One problem with these linear networks is overfitting. During training, randomly removed units and their connections are included in the neural network’s dropout. This mitigates overfitting by keeping the model from forming intricate co-adaptations on the training set. The dropout in the model is 0.2.

3. Adaptive learning rate: It was discovered that 1 ×10⁻² was the ideal initial learning rate for the model. However, after extensive testing in later epochs, it was found that, on the test dataset, when the model’s performance peaked, it rapidly overfitted the training dataset. To counteract this, a constant factor periodically decreases the learning rate after a predetermined number of epochs. Once every two epochs, the model was added with a step decay factor of 0.25 to keep the model from overfitting the dataset.

Performance across length

Figure 5 displays the F1-scores and the exact match (EM) for answers up to 15 tokens so that you can assess how well the algorithm performed while predicting responses of varying durations. This proposed model anticipates that the model’s performance will deteriorate with increasing answer length, as is the case with many NLP applications, including neural machine translation. Nevertheless, for answers up to length 12, there is no discernible decline in the F1-score. However, the model observes a steady decrease in accuracy for both EM after that. This makes sense intuitively since the number of words increases, and the computation of the proper answer span gets harder. The graph shows that the EM and F1 scores decrease as response lengths do. Furthermore, this work notes that when the answer length increases, the difference between F1, EM, and accuracy drops at distinct rates.

Performance across questions with non-interrogative head

This work could not locate any research that looked at the system’s performance for questions that started with non-interrogative words, even though it is usual for current literature to analyze performance across questions beginning with different interrogative words. These questions are typically more sophisticated and rhetorical and are called non-interrogative. This article examined how well the proposed model performed for these questions, and Fig. 7 displays the results for several of them. This work finds that this model’s effectiveness varies significantly across different start words, but this study can identify some intriguing events. Figs. 8 and 9 illustrates how effectively this model works for questions that begin with “about” and “approximately.” The stark contrast between the F1-score and EM for questions that start with “although” is the most intriguing finding. This work may easily anticipate that questions beginning with “although” will be structurally complicated sentences based on how this research uses the language daily. Based on the performance of this model on questions of this type, the model can observe that while it is challenging to obtain the precise answer in this instance, this model performs remarkably well in getting the general idea of the solution. This work believes that further research on enhancing performance for non-interrogative questions will result in systems that can comprehend natural language’s complex meaning. Ultimately, the model obtained the following outcomes: “Test” denotes the hidden test set kept up to date by ‘y’, the SQuAD2.0 creators.

F1 and EM scores

Unfortunately, the performance of this BiDAF model was not up to pace. The final F1 and EM scores for the training set were approximately 10% and 6%, respectively. For EM and F1, the approximate final validation scores were 3.7% and 7.6%, respectively. The outcomes outperform this model’s initial baseline. Plots of the data collected during the four training epochs are shown above. According to this study, the model’s average loss and global norm decrease slowly while its F1 and EM scores rise shown in Table 3. Given the low learning rate in these early epochs, the proposed model F1 and EM scores may move slowly. The model in this work was trained using a fixed learning rate of 0.0001. This article should have started with a more significant learning rate, especially during the first epoch when the model does not have to care about fine-grained learning, even though a low learning rate is best for later epochs. Although tinkering with the initial value and decay rate would have taken a lot of time, having an exponentially declining learning rate would have been nice. This model’s sluggish development was also caused by the massive number of parameters it had to learn.

This is why the research developed word vectors that effectively capture syntactic and semantic information using trainable 300d GloVe vectors. It sounds okay, but the 41 m parameters the model had to learn were significantly increased. Due to time constraints, it is only possible to train for four epochs, which took roughly eleven hours each. This model was learning appropriately based on the global norm and diminishing loss of this work, in addition to its F1/EM scores. This work would expect these measurements to be relatively low once the model has completed training and the optimal set of parameters has been identified. Nevertheless, considering the concave F1 and EM curves, the model predicts that the scores would not increase significantly. This can point to a severe issue with the model or a flaw in the test code. For a reasonable model to overfit the training set and obtain noticeably greater training accuracy, tens of millions of parameters should be plenty.

Time performance

The most significant challenge to testing, refining, and training the models proposed in this study was time. Testing minor or hyperparameter adjustments was particularly challenging because the recommended BiDAF model required more than 10.5 h per GPU epoch to train. They asserted that forgetting errors constituted the primary source of this research’s limitations. A batch size more significant than four resulted in out-of-memory issues that may manifest hours into training on the forty million parameter model utilized in this work. Looking back, this study should have run a much larger batch using smaller GloVe vectors or restricting the context paragraph size to a few hundred. An evaluation of the model’s bidirectional dynamic RNNs with the setting “Swap Memory = True” revealed no discernible effect on memory efficiency. The model also carefully examined test code to hunt for explicit for-loops that might be utilizing excessive amounts of memory or carrying out tensor calculations inefficiently; nevertheless, nothing unusual was found. The attention flow layer, which is the core of the BiDAF model, also significantly prolonged the training duration. This layer consumed many processing resources since it created the attention layer’s similarity matrix from the 200-dimensional hidden states of every word in the question and context paragraph.

The used SQuAD2.0 (Stanford Question Answering Dataset) benchmark dataset for evaluating machine comprehension and question answering systems. Unlike its predecessor, SQuAD1.1, SQuAD2.0 incorporates unanswerable questions, adding a layer of complexity that better reflects real-world scenarios. This enhancement encourages models to not only identify answers within a given context but also recognize when questions cannot be answered, fostering more robust and accurate performance evaluation. By including unanswerable questions, SQuAD2.0 promotes the development of AI systems capable of discerning between answerable and unanswerable queries, thus advancing the state-of-the-art in natural language understanding and question answering. In future, we plan to validate the performance of the proposed model on large scale real time dataset.