AMPLIFY: attention-based mixup for performance improvement and label smoothing in transformer

Introduction

Data augmentation techniques are widely used in modern machine learning field to improve the predictive performance and robustness of computer vision and natural language processing (NLP) models by adding feature noise to the original samples. Compared to computer vision tasks, NLP tasks face samples with more complex data structures, semantic features, and semantic correlations, as natural language is the main channel of human communication and reflection of human thinking (Li, Hou & Che, 2022). Therefore, NLP models are more sensitive to the quality of the dataset, and often require a variety of data augmentation techniques to improve the model’s generalization ability, adaptability, and robustness to new data in practical engineering applications. Common text data augmentation techniques include random synonym replacement (Zhang, Zhao & LeCun, 2015), back-translation (Xie et al., 2020), random word insertion and deletion, etc. (Wei & Zou, 2019). The core idea of these methods is to generate more training samples by performing a series of feature transformations on the original samples, allowing the model to learn and adapt to changing feature information, and enabling the model to better deal with and process uncertainty in the samples. Additionally, by selectively increasing the size of the training set, data augmentation techniques can effectively alleviate the negative impact of limited data and imbalanced data classification on model prediction performance.

Standard mixup is a simple and effective image augmentation method first proposed in 2017 (Zhang et al., 2017). It aggregates two images and their corresponding labels by linear interpolation to generate a new augmented image and its pseudo-label. The main advantage of the standard mixup technique is that it purposefully generates beneficial noise by using the weighted average of features in the original samples. After adapting to this noise, the model becomes less sensitive to other noise in the training samples, thereby improving the model’s generalization ability and robustness. Additionally, because it can generate more augmented samples by aggregating different original sample pairs, even at a high data augmentation magnitude, there is no duplicate aggregation result. This greatly enhances the diversity of augmented samples, thereby effectively improving the training efficiency of the model and reducing the risk of overfitting. In the NLP field, the standard mixup technique can be broadly divided into two categories: Input Mixup (Yun et al., 2019) and Hidden Mixup (Verma et al., 2019). Input Mixup increases the number of samples in the training set input to the model. The specific process involves padding all text sequences in the training dataset to the same length. After completing word embedding processing, random pairs of sequences are combined. Mixup operations are then applied to the two vectorized samples in each sequence pair, along with their corresponding classification labels, generating augmented samples and their pseudo-labels. Finally, the augmented samples and pseudo-labels are fed into the model for training. Input Mixup performs mixup operations during the word embedding stage of text preprocessing and only involves semantic features in the word vector space. Its main objective is to increase the training set size and reduce overfitting. Hidden Mixup applies mixup operations in the hidden layers of the model. The specific process involves randomly selecting two sequences, x and x′, from the training set. After processing through the model, a random layer is selected from the model’s hidden layers, and the output features, h and h′, of x and x′ at that layer are extracted. Mixup is then performed on h, h′, and their labels, resulting in linearly interpolated mixed features and corresponding pseudo-labels.

$\overline{h}=\lambda h+\left(1-\lambda \right)h^{\prime }$.

Finally, the mixed features are fed into subsequent hidden layers. Hidden Mixup performs mixup during the feature extraction stage of text processing and only involves specific hidden layers in the model. Its main objective is to enhance the model’s generalization ability without requiring additional input data. In summary, Input Mixup increases the sample quantity, while Hidden Mixup increases feature diversity. Both methods focus on improving model performance through mixup operations, albeit with different emphases. Verma et al. (2019)’s research has shown that performing mixup operations at the input layer of the model leads to underfitting, while performing mixup at deeper layers makes the training easier to converge. However, when the original samples contain noise or outliers, performing mixup may result in generated augmented samples containing more regular noise or outliers due to the principle of linear interpolation. If the model learns too much from this noise and outliers, it may cause obvious overfitting and prediction bias, thereby reducing the model’s generalization performance. The basic idea of Standard Mixup is to linearly aggregate two samples $\left(x_i,y_i\right)$ and $\left(x_j,y_j\right)$ in the training set 𝔻_train, to generate a new sample $\left(x_{\mathrm{mix}},y_{\mathrm{mix}}\right)$ as an augmented input to train the network, where x represents the sample and y represents its corresponding label. This aggregation process can be expressed by the following formulae:

x_mix = λx_i + (1 − λ)x_j

y_mix = λy_i + (1 − λ)y_j

where λ is the weight sampled from the Beta distribution.

Multi-head attention (MHA) (Vaswani et al., 2017) is a technique commonly used in sequence-to-sequence models to calculate feature correlations. Namely, it can calculate the attention weights for each position in the sequence in parallel using different attention heads, and then obtain the representation of the entire sequence by summing the weighted values. Specifically, MHA helps the model establish correspondences between features in the input and output sequences, thereby improving the expressiveness and accuracy of the model. It can also attend to different features at different positions in the input sequence without taking into account their distance, assigning different weights to different parts of the input to capture information relevant to the output sequence. To address the problem of noise and out-of-distribution values in original samples, which affect the aggregated samples in standard mixup, our AMPLIFY method uses the Hidden Mixup idea to perform mixup operations on the hidden layers of the Transformer block. Since the output of the MHA already includes the model’s attention to different parts of the input sequence, it can guide the model on which features to retain and which to discard when aggregating different samples, ultimately integrating more critical features into the generated output sequence. Therefore, AMPLIFY duplicates the MHA output results of the sample sequence in the same batch, shuffles the order of these copies, and then performs mixup on them with the original output results. This allows the model to aggregate the correlations and attention of two sample features multiple times at different levels, thereby obtaining more reasonable weights for each position in the aggregated sequence and avoiding unnecessary noise or feature information loss during the mixup process. Additionally, since the attention mechanism itself is a method of weighting the input features, AMPLIFY is more natural and effective intuitively.The overall structure of the model is depicted in Fig. 1.

Since AMPLIFY aggregates the features of the augmented sample and the corresponding original sample in the hidden layers of the model, it has lower computational cost and fewer parameters compared to other data augmentation methods. It also avoids the problem of increased time and resource consumption caused by standard mixup methods. In addition, Our result have shown that our method can better learn the key features of the input sequence, improving the generalization ability and prediction performance of the model. For example, on the MRPC dataset, AMPLIFY’s average accuracy is 1.83% higher than the baseline, and 1.04%, 1.72%, and 1.46% higher than EmbedMix, SentenceMix, and TMix, respectively.

Related Work

Method

Portions of this text were previously published as part of a preprint (https://arxiv.org/abs/2309.12689)

Assuming that our AMPLIFY method requires the following inputs, outputs, function definitions, and data structures:

• T_pre is a pre-trained text classification model based on Transformer architecture.

• ${\mathbb{D}}_{train}={\left\{〈x_i,y_i〉\right\}}_{i=1}^m$ is the training set of a downstream text classification task with m samples, where x_i is a sample sequence and y_i is its corresponding label.

• ${\mathbf{M}}_B^o=\left\{〈x_1^o,y_1^o〉,〈x_2^o,y_2^o〉,\dots ,〈x_l^o,y_l^o〉\right\}$ is the mini-batch of T_pre during each training iteration, with a sample size of l.

• $S\left({\mathbf{M}}_B^o\right)$ is a random shuffling function that is responsible for changing the order of samples in the mini-batch.

• $I\left({\mathbf{M}}_B^o\right)$ is an index function that is responsible for retrieving a collection of text sequences ${\mathbf{M}}_B^o$ with l elements and returning the corresponding index according to the order of these elements.

• $R\left({\mathbf{M}}_B^o,inde{x}_R\right)$ is a reordering function that is responsible for reordering the elements in the collection of text sequences ${\mathbf{M}}_B^o$ with l elements according to the index index_R, and returning the sorted sequence collection.

• $A_{\text{pre}}^{\text{MHA}}\left(x\right)$ represents the corresponding feature sequence output after the input sample sequence x is fed forward to the Muti-Head Attention layer in the Transformer block of T_pre.

• $\text{Beta}\left(\alpha ,\alpha \right)$ represents the U-shaped Beta probability distribution function with shape parameter α.

• $\text{WS}\left(\text{Beta}\left(\alpha ,\alpha \right)\right)$ represents a weight value obtained by sampling according to the U-shaped beta probability distribution.

• $\text{LABEL}\left({\mathbf{M}}_B^o,inde{x}_R\right)$ is a label generation function that selects the corresponding labels from a collection of sequences ${\mathbf{M}}_B^o$ in the order provided by the index index_R and puts them together into a label set.

• ${\mathcal{F}}_N\left({\mathbf{M}}_B^o\right)$ is the prediction function of the text classification model, which generates a set of probabilities for each sequence corresponding to N categories based on the input sequence set ${\mathbf{M}}_B^o$.

Based on the above initial conditions, our AMPLIFY algorithm includes the following specific steps:

• Step 1: When model T_pre is trained for downstream text classification tasks, a corresponding mini-batch, referred to as ${\mathbf{M}}_B^o$, is obtained from training set 𝔻_train through the following formula in each iteration: (1)${\displaystyle {\mathbf{M}}_B^o:{\mathbb{D}}_{train}\to {\left\{〈x_i^o,y_i^o〉\right\}}_{i=1}^l=\left\{〈x_1^o,y_1^o〉,〈x_2^o,y_2^o〉,\dots ,〈x_l^o,y_l^o〉\right\}}$

• Step 2: A copy of ${\mathbf{M}}_B^o$ is made and named ${\mathbf{M}}_B^s$. The sample sequences in ${\mathbf{M}}_B^s$ are shuffled randomly, and then the index of the shuffled elements is obtained through the indexing function $I\left({\mathbf{M}}_B^s\right)$, which is denoted as index_R (it is used to calculate the loss value, as detailed in Eq. (8)). This process can be expressed as the following formulae: (2)${\displaystyle {\mathbf{M}}_B^o\to {\mathbf{M}}_B^s={\left\{〈x_i^s,y_i^s〉\right\}}_{i=1}^l=\left\{〈x_1^s,y_1^s〉,〈x_2^s,y_2^s〉,\dots ,〈x_l^s,y_l^s〉\right\}inde{x}_R=I\left(S\left({\mathbf{M}}_B^s\right)\right)}$

• Step 3: Calculate the value of $A_{\text{pre}}^{\text{MHA}}\left(x_i^o\right)$ for each sample sequence $x_i^o$ in ${\mathbf{M}}_B^o$, and obtain the corresponding feature sequence set for ${\mathbf{M}}_B^o$, denoted as ${\mathbf{F}}_B^o$. This process can be expressed as the following formula: (3)${\displaystyle {\mathbf{F}}_B^o=\left\{〈A_{\text{pre}}^{\text{MHA}}\left(x_1^o\right),y_1^o〉,〈A_{\text{pre}}^{\text{MHA}}\left(x_2^o\right),y_2^o〉,\dots ,〈A_{\text{pre}}^{\text{MHA}}\left(x_l^o\right),y_l^o〉\right\}}$

• Step 4: Make a copy of ${\mathbf{F}}_B^o$ called ${\mathbf{F}}_B^s$, and reorder the sequence label pairs in ${\mathbf{F}}_B^s$ based on index_R using funciton $R\left({\mathbf{F}}_B^o,inde{x}_R\right)$. The process can be represented by the following equations: (4)${\displaystyle {\mathbf{F}}_B^o\to {\mathbf{F}}_B^s=R\left({\left\{〈A_{\text{pre}}^{\text{MHA}}\left(x_i^o\right),y_i^o〉\right\}}_{i=1}^l,inde{x}_R\right)=\left\{〈A_{\text{pre}}^{\text{MHA}}\left(x_1^s\right),y_1^s〉,〈A_{\text{pre}}^{\text{MHA}}\left(x_2^s\right),y_2^s〉,\dots ,〈A_{\text{pre}}^{\text{MHA}}\left(x_l^s\right),y_l^s〉\right\}.}$

• Step 5: Perform element-wise mixup operation on ${\mathbf{F}}_B^o$ and ${\mathbf{F}}_B^s$ to obtain the aggregated feature sequence set ${\mathbf{M}}_B^{\text{mix}}$, which is then fed into the subsequent hidden neural network layer. Considering that AMPLIFY requires mixup operation to be performed in each Transformer block, if the weight coefficients λ for mixup in different blocks are not the same, it will lead to frequent and intense disturbance to the features in the sequence, and some abnormal features or noise will also be constantly accumulated and strengthened, resulting in significant fluctuations in the model’s loss value during the actual experimental process. Moreover, when there is a large difference in the λ values between each block, this instability will be further amplified. Therefore, the λ_max we use is different from the definition in the standard mixup. To improve the stability of linear interpolation and the consistency of feature representation, in the AMPLIFY algorithm, we adopt a method of first performing multiple samplings based on the Beta probability distribution (BPD), and then selecting the maximum value from the resulting λ values. Specifically, we call function $\text{Beta}\left(\alpha ,\alpha \right)$ multiple times to obtain a weight set with n elements, denoted as Λ, and then select the maximum weight value λ_max in this set as the weight coefficient for all mixup operations in the model. This method can significantly reduce the adverse impact of randomness in the feature sequence on linear interpolation, making one feature sequence the explanatory term and the other the random perturbation term, which is also the reason why we tend to choose smaller α values. In addition, according to Zhang et al. (2017)’s research, when α → ∞, the value of the BPD will approach 0.5 infinitely, and the training error of the model on the real dataset will also increase significantly. If the standard mixup calculation method is used for mixing operation, it is likely to produce a large number of abnormal features due to significant disturbance caused by linear interpolation, which, in turn, will make the model prediction to deviate and greatly reduce its generalization performance. On the other hand, as shown in Fig. 2, since the weight value obtained by randomly sampling only once from the U-shaped BPD may fall into the low probability area, or even be very close to Therefore, sampling a total of n weights from the BPD and taking the maximum value can effectively avoid this issue. According to experimental results, we also found that if the value of n is large, the weight values sampled from the U-shaped BPD will appear in the high probability area in large quantities, and the maximum value selected from them will also be closer to 1. This results in a significant reduction in the weight of the feature sequence ${\mathbf{F}}_B^o$ or ${\mathbf{F}}_B^s$ that serves as the random perturbation term, and even renders it ineffective as a perturbation, thus making mixup meaningless. Eventually, based on the experimental results Table 1 (we compared the classification performance of AMPLIFY with different weight sampling numbers on five datasets. The mean in the table is the average performance on the five datasets, and it reaches its optimal when sampled five times), we choose a = 0.1 and n = 5 to avoid obtaining risky weight values as much as possible and effectively leverage the role of random perturbation term. The above process can be described by the following equations: (5)${\displaystyle \Lambda ={\left\{{\lambda }_i=\text{WS}\left(\text{Beta}\left(\alpha ,\alpha \right)\right)\right\}}_{i=1}^n=\left\{{\lambda }_1,{\lambda }_2,\dots ,{\lambda }_n\right\},{\lambda }_{\text{max}}=\text{max}\left(\Lambda \right).}$ (6)${\displaystyle x_i^{\text{mix}}={\lambda }_{\text{max}}\cdot x_i^o+\left(1-{\lambda }_{\text{max}}\right)\cdot x_i^s{y}_i^{\text{mix}}={\lambda }_{\text{max}}\cdot y_i^o+\left(1-{\lambda }_{\text{max}}\right)\cdot y_i^s{\mathbf{M}}_B^{\text{mix}}={\left\{〈x_i^{\text{mix}},y_i^{\text{mix}}〉\right\}}_{i=1}^l=\left\{〈x_1^{\text{mix}},y_1^{\text{mix}}〉,〈x_2^{\text{mix}},y_2^{\text{mix}}〉,\dots ,〈x_l^{\text{mix}},y_l^{\text{mix}}〉\right\}.}$

• Step 6: Calculate the loss value ${\mathcal{L}}_{\text{mix}}$ based on the predicted results of the model. Standard Mixup uses two common methods to calculate the loss value. One method calculates the cross-entropy loss value by taking the logits output by the text classification head and computing it with the ground truth in both the original order and the shuffled order, and then weighting the two loss values and adding them together as the total loss. The other method calculates the cross-entropy loss value between the logits and the mixed pseudo-labels. Essentially, the main difference between these two methods is the order in which the weighting and cross-entropy loss calculations are performed. The first method calculates the cross-entropy loss value of the results first, and then performs weighting and summation, while the second method first performs weighting and summation of the results before calculating the corresponding cross-entropy loss value. In the AMPLIFY algorithm, according to the experimental results of Yoon, Kim & Park (2021), we adopt the more effective first method to calculate the loss value ${\mathcal{L}}_{\text{mix}}$. The specific process can be represented by the following equations: (7)${\displaystyle logits={\mathcal{F}}_N\left({\mathbf{M}}_B^{\text{mix}}\right),inde{x}_o=I\left({\mathbf{M}}_B^o\right).}$ (8)${\displaystyle g{t}_o=\text{LABEL}\left({\mathbf{M}}_B^o,inde{x}_o\right),g{t}_s=\text{LABEL}\left({\mathbf{M}}_B^o,inde{x}_R\right){\mathcal{L}}_{\text{mix}}\Leftarrow {\lambda }_{\text{max}}\cdot CrossEntropy\left(logits,g{t}_o\right)+\left(1-{\lambda }_{\text{max}}\right)\cdot CrossEntropy\left(logits,g{t}_s\right).}$

From the perspective of reflecting the correlation between labels, the method of mixing labels in the AMPLIFY algorithm can be considered as an enhanced version of label smoothing. Through the weight coefficient λ_max, we can determine how much proportion of the cross-entropy loss value comes from the interpretive term and how much proportion comes from the random perturbation term. This is equivalent to adding moderate noise to the original labels, so that the modelś prediction results do not overly concentrate on the categories with high probabilities, leaving some possibilities to the categories with low probabilities, while effectively reducing the risk of overfitting. In summary, the pseudocode of the AMPLIFY algorithm is shown as follows:

 
_______________________ 
 Algorithm 1: Algorithm of AMPLIFY.                                       _________ 
    Input: 
    The pre-trained text classification model based on Transformer architecture 
     Tpre 
      The training dataset of downstream text classification task with m samples 
     Dtrain = {〈xi,yi〉}m 
             i=1 
      The mini-batch during each training iteration 
     Mo 
        B = {〈xo 
1,yo 
1〉,〈xo 
2,yo 
2〉,...,〈xo 
l ,yo 
l 〉} 
  The weight coefficient for all Mixup operations in the model λmax 
      The random shuffling function S (Mo 
     B) 
     The U-shaped Beta probability distribution function Beta (α,α) 
     The feature sequence output by the Muti-Head Attention layer AMHA 
pre    (x) 
     Output: 
    The Logits output by the text classification head logits 
    The final loss value of the model Lmix 
      foreach Mo 
    B ⊂ Dtrain do 
         Use equation ?? to obtain the shuffled mini-batch Ms 
    B and its index 
     indexR. 
   foreach multi-head attention layer ∈ Tpre do 
       Use equation ?? to obtain the feature sequence set Fo 
   B corresponding 
to Mo 
    B. 
   Use equation ?? to obtain the re-ordered feature sequence Fs 
   B based 
on indexR. 
   Use equations 6 to element-wise mixup Fo 
   B and Fs 
   B, obtaining the 
aggregated feature sequence set Mmix  
     B  . 
   end 
   Use equation ?? to obtain the prediction result logits of the model. 
   Use equations ?? to calculate the loss value Lmix 
      end    

Experiments

Results

Overall results

Table 2 details the impact of our AMPLIFY method and other standard mixup methods on the performance of baseline pre-trained text models (denoted as “No Mixup” in the table) on seven benchmark datasets. The results show that AMPLIFY provides better performance gains for the baseline model, and the idea of introducing a random perturbation term also serves as a good regularization to reduce the risk of overfitting. Moreover, with the continuous iteration of the training process, AMPLIFY almost outperforms other standard methods in all experiments, especially on the TREC-Fine dataset, where its improvement on model accuracy reaches 4.2%. In addition, in terms of the variance of the results, the performance gain of AMPLIFY is relatively stable, indicating that it has good robustness to deal with uncertainty in the samples, as well as better overfitting resistance than other mixup methods.

Table 2:

Comparison experimental results of different mixup methods on seven benchmark datasets.

All values in the table are the average accuracy (%) and variance of the model after running three times with three different random seeds Dror et al. (2018). The bold indicates the optimal results in each dataset.

Method		GLUE		TREC		SetFit		IMDB
	MRPC	SST-2	coarse	fine	SST-5	YELP-5
No mixup	81.20  ± 0.442	91.05  ± 0.605	96.20  ± 0.240	86.47  ± 0.596	53.24  ± 0.519	97.18  ± 0.006	91.45  ± 0.007
EmbedMix	81.99  ± 0.346	91.31  ± 0.029	96.73  ± 0.036	89.67  ± 0.809	51.98  ± 0.306	97.15  ± 0.008	91.61  ± 0.008
SentenceMix	81.31  ± 0.135	91.71  ± 0.097	96.40  ± 0.187	89.87  ± 1.316	53.05  ± 0.017	97.12  ± 0.002	91.54 ± 0.024
TMix	81.57  ± 0.271	91.62  ± 0.065	96.87  ± 0.062	89.67  ± 0.969	52.91  ± 0.246	97.13  ± 0.003	91.54  ± 0.005
Ours	83.03  ± 0.110	91.01  ± 0.041	97.00  ± 0.187	90.67  ± 0.276	53.41  ± 0.227	97.18  ± 0.001	91.63  ± 0.009

DOI: 10.7717/peerjcs.2011/table-2

What caught our attention is that on the Yelp-5 dataset with 560,000 training samples, almost all mixup methods failed to provide a net performance gain for the baseline model. Observing the experimental process and results, We believe this is because pre-trained models like BERT generally perform well when facing large-scale datasets, as they are usually pre-trained on massive corpora of textual data, allowing them to learn more language-level knowledge and patterns. Therefore, they have already achieved excellent performance on datasets such as Yelp-5, which have relatively clear classification features, leaving limited room for mixup to improve their performance. On the other hand, mixup methods essentially augment the samples and their labels by using the existing feature information in the same dataset, which is different from standard data augmentation strategies and does not introduce out-of-domain feature information. Therefore, they cannot significantly improve the model’s performance or seriously impair it. When the dataset is large, the baseline model can already understand the text features well. In this case, using mixup methods may not have a significant effect. However, when the dataset is small and the model’s overfitting problem is severe, like many other data augmentation methods, mixup can often have a significant effect, helping the model improve its generalization ability and robustness in the face of sample uncertainty (Sun et al., 2020).

For example, the MRPC dataset has only 4,500 samples, and using the mixup method on it can lead to performance gains, especially for AMPLIFY, which performs multiple mixup operations during the model’s prediction process, effectively adding multiple mild random perturbations to the feature sequence, resulting in more significant performance gains. Additionally, when using the mixup method to augment feature sequences, the mixed sequences must differ significantly from the original ones to improve the model’s generalization performance. If the number of sequences per class in the dataset is relatively balanced, the distribution of differences between the mixed samples will also be more uniform, making it difficult for the mixup method to bring further performance gains, as demonstrated in the study by Yoon, Kim & Park (2021). From this perspective, the sample sizes of various categories in Yelp-5 are very similar, and the feature distribution of the data is relatively balanced. Therefore, the differences between the mixed sequences are not significant enough, which renders the mixup method ineffective. Furthermore, on SST-2, the AMPLIFY method resulted in negative gains in model performance. After analyzing the reasons, we found that SST-2 is a dataset used for sentiment binary classification tasks, and it contains text samples of audience comments on movies or annotations of audience sentiment on movies. Therefore, these comment texts vary greatly in length, and after padding, many text sequences contain a large number of meaningless placeholders. Therefore, when performing mixup operations on feature sequences of short and long texts, it may mix placeholders with sentiment information, which weakens or even submerges the classification features, thereby affecting the model’s prediction results.

To further validate the effectiveness of AMPLIFY, we selected the distilled version of BERT, DistilBERT, as the backbone network for comparison. Table 3 details the impact of AMPLIFY on the performance of DistilBERT on seven benchmark datasets. As can be seen from the results, since DistilBERT has only six Transformer encoder blocks compared to BERT’s 12, the number of mixup operations on DistilBERT with AMPLIFY was reduced by half, resulting in a less significant improvement in performance compared to the standard BERT model. However, consistent performance net gains were still achieved, indicating that AMPLIFY can have a greater impact on complex Transformer models.

Table 3:

The effectiveness of using AMPLIFY on DistilBERT.

The bold indicates the optimal results in each dataset.

Method		GLUE		TREC		SetFit		IMDB
	MRPC	SST-2	coarse	fine	SST-5	YELP-5
no mixup	81.76 ± 0.507	90.17 ± 0.135	97.13 ± 0.222	85.39 ± 1.307	50.99 ± 0.955	96.75 ± 0.005	90.62 ± 0.003
EmbedMix	81.79 ± 0.346	90.11 ± 0.029	97.03 ± 0.036	85.67 ± 0.809	50.98 ± 0.306	96.75 ± 0.008	90.61 ± 0.008
SentenceMix	81.74 ± 0.385	90.18 ± 0.101	97.00 ± 0.001	85.80 ± 1.386	50.83 ± 0.991	96.69 ± 0.011	90.70 ± 0.014
TMix	82.12 ± 0.216	89.62 ± 0.127	96.67 ± 0.249	85.73 ± 0.436	49.85 ± 0.088	96.73 ± 0.002	90.82 ± 0.015
Ours	81.87 ± 0.121	90.17 ± 0.062	97.13 ± 0.115	85.80 ± 0.779	51.07 ± 0.645	96.80 ± 0.005	90.64 ± 0.003

DOI: 10.7717/peerjcs.2011/table-3

Figure 3 illustrates a heatmap of the p-values obtained from t-tests comparing our method with other methods for the attention output matrices in each dataset we computed. Each cell represents the t-test p-value between our method and other methods. It is worth noting that in this heatmap, the p-values between our method and all other mixup methods are less than 0.05, indicating significant differences. These statistically significant differences should not be overlooked. They suggest that our method exhibits notable distinctions from other mixup methods in generating attention matrices, further emphasizing the superiority of our approach. By analyzing these significant differences, we can gain a deeper understanding of the improvements our method brings to attention output and quantify their statistical significance. Firstly, we can discuss the quality of the attention matrices. Through the attention analysis, our method demonstrates superior performance in terms of stability, consistency, and accuracy compared to other methods. This implies that our method is better able to capture crucial features and allocate attention weights more accurately. Furthermore, the impact of these significant differences on practical tasks is also noteworthy. Experimental results indicate that our method exhibits better performance in specific tasks. Such practical performance disparities further strengthen the advantages resulting from the improvements in attention output brought about by our method. Additionally, a notable aspect of our method among mixup approaches lies in how it leverages attention mechanisms to enhance the effectiveness of mixup. This uniqueness positions our method as more advantageous in attention output compared to other methods and further enhances the overall performance.

To further validate the potential issue of mixup propagating noise or outlier features from the original samples to the augmented samples, leading to over-sensitivity of the model, as well as to demonstrate the effectiveness of our method in improving generalization and denoising, we conducted additional experiments on five different datasets by randomly deleting or swapping original data at proportions of 5%, 10%, 15%, and 20%. These experiments aimed to investigate the performance of different mixup methods in the presence of noisy datasets. The results in Tables 4 and 5 demonstrate that our method achieved the highest accuracy in almost all datasets. We attribute this success to the introduction of attention mechanisms in our method, which allows the model to focus on important features and adaptively adjust attention weights during the mixup process. This adaptive adjustment of attention weights enables our method to handle variations introduced by randomly deleting or swapping data samples more effectively. By dynamically allocating attention weights, the model can effectively handle missing or swapped data, resulting in more accurate predictions. Furthermore, the performance improvement of our method, AMPLIFY, can also be attributed to its utilization of inherent dependencies and correlations within the data through attention mechanisms. By attending to relevant regions or features, AMPLIFY can effectively propagate useful information among samples, thereby enhancing generalization and accuracy.

Table 4:

Results of four mixup methods on SST-2, SST-5, MRPC, TREC-coarse and TREC-fine training sets with random deleted of data at various proportions of 5%, 10%, 15%, and 20%.

The bold indicates the optimal results in each dataset.

Dataset	Percentage	No Mixup	EmbedMix	SentenceMix	TMix	Ours
MRPC	5%	68.52	67.30	68.41	68.46	73.04
	10%	67.83	67.25	66.84	67.77	72.87
	15%	67.30	67.30	68.23	67.19	72.64
	20%	67.25	68.99	68.93	68.81	71.65
SST-5	5%	39.55	37.87	40.63	39.19	44.30
	10%	37.96	36.02	37.56	36.61	43.12
	15%	39.23	37.60	40.50	39.32	43.26
	20%	34.71	35.43	38.64	36.61	42.62
SST-2	5%	85.67	86.60	86.82	86.99	87.48
	10%	86.38	86.00	86.44	86.44	86.66
	15%	84.68	83.96	85.28	85.61	87.48
	20%	83.53	83.03	82.87	83.47	85.61
IMDB	5%	91.49	91.51	91.56	91.64	91.78
	10%	91.55	91.72	91.61	91.67	91.89
	15%	91.67	91.64	91.57	91.42	91.81
	20%	91.45	91.50	91.55	91.60	91.65
TREC-coarse	5%	95.80	96.60	96.40	96.20	96.80
	10%	96.80	96.40	96.20	95.20	97.00
	15%	96.40	95.80	95.40	96.40	96.60
	20%	96.00	95.80	96.10	95.90	96.40
TREC-fine	5%	87.00	85.80	87.60	86.20	87.80
	10%	86.40	86.20	87.00	87.20	87.60
	15%	85.20	85.40	84.40	84.40	85.60
	20%	84.60	84.20	84.20	84.80	85.80

DOI: 10.7717/peerjcs.2011/table-4

Table 5:

Results of four mixup methods on SST-2, SST-5, MRPC, TREC-coarse and TREC-fine training sets with random swapped of data at various proportions of 5%, 10%, 15%, and 20%.

The bold indicates the optimal results in each dataset.

Dataset	Percentage	No Mixup	EmbedMix	SentenceMix	TMix	Ours
MRPC	5%	67.88	66.90	68.75	67.01	72.93
	10%	67.65	64.35	67.19	66.67	71.30
	15%	66.20	65.16	63.01	59.94	70.72
	20%	65.10	65.04	63.13	66.90	67.30
SST-5	5%	40.65	40.95	40.95	41.95	43.67
	10%	36.86	35.88	34.98	37.06	44.25
	15%	38.16	38.55	37.96	38.14	41.36
	20%	36.60	22.04	35.20	38.64	38.96
SST-2	5%	81.76	85.94	87.31	86.11	87.59
	10%	84.97	85.17	84.46	85.34	87.77
	15%	84.88	83.42	84.68	85.50	87.59
	20%	85.39	82.59	82.92	82.04	87.15
IMDB	5%	91.42	91.46	91.40	91.56	91.74
	10%	91.43	91.43	90.96	90.79	91.56
	15%	90.25	89.82	90.36	90.28	90.56
	20%	86.51	88.80	88.48	88.35	89.08
TREC-coarse	5%	97.00	96.40	97.00	96.60	97.80
	10%	95.80	96.60	96.20	96.20	96.80
	15%	94.60	95.60	95.60	95.60	95.80
	20%	96.20	96.00	95.60	95.40	96.80
TREC-fine	5%	86.60	86.40	86.40	87.40	87.60
	10%	85.80	84.80	86.00	84.80	86.40
	15%	84.60	84.00	84.40	84.60	85.80
	20%	84.80	85.00	86.00	83.60	85.80

DOI: 10.7717/peerjcs.2011/table-5

Visualization of experimental results

Figure 4 shows the cross-entropy loss values of four mixup methods, EmbedMix, TMix, AMPLIFY, and SentenceMix, on the MRPC dataset for the first 12k training iterations. From this figure, it can be seen that the AMPLIFY method has a lower loss value and less fluctuation during the training process compared to other mixup methods, indicating a more stable training process. However, using mixup operation may aggregate noise and outlier features from the original sequence into the mixed sequence, leading to overly concerning these interference information and reducing the model’s generalization ability. Specifically, the mixed sequence contains features from both two original sequences, but the linear interpolation operation also interferes with the information from them, weakening or even drowning out useful features. Therefore, as a special data augmentation technique, mixup increases noise during the training process, making it harder for the model to fit the data, and thus increasing the loss value. As a comparison, AMPLIFY reduces interference from noise and useless features by adding mild random perturbation terms to the explanatory terms multiple times, while retaining the advantages of the mixup method. In terms of computation time, the experimental results further illustrate that AMPLIFY saves 24s, 32s, and 472s compared to EmbedMix, SentenceMix, and TMix, respectively, within the first 12k iterations. It is shown that while bringing better performance gains and more stable performance, AMPLIFY also largely saves the computational cost of other mixup methods during the mixing process.

Figure 5 shows the effect of the two different mixup operations on the attention mechanism in the same text sequence. It is clear that after the AMPLIFY operation, the attention between words in the same sentence remains at a high level, while after the EmbedMix operation, the MHA cannot recognize the special separator between the two sentences very well, and establishes a high-level attention between words in different sentences. This is because the linear interpolation operation of EmbedMix makes the feature sequence of a mixed sentence contain context information from another sentence, which even overwhelms the semantics of the separator itself, causing confusion in the understanding of sentence context by the attention mechanism.

Figure 6 shows the effect of the two mixup operations on the attention mechanism when applied to a specific word in the same sentence. Clearly, AMPLIFY does not have a negative impact on the output of MHA, the separator is successfully recognized, and closely related words still have a high attention weight, such as the two words “rabbit” and “hopped” which have the highest attention weight. In contrast, EmbedMix has a negative impact on the output of MHA, making it difficult to select the correct word and assign appropriate attention weights, resulting in the separator not playing its role and the attention being scattered.

Figure 7 demonstrates the impact of different mixup operations on MHA when querying other words in the same sentence that have high correlation with the word “it”. The query vector q and key vector k jointly determine the correlation value between any two words, while the element-wise multiplication of q and k, q⋅k, determines the attention value, and the softmax provides the query result based on the attention distribution. In the figure, EmbedMix focuses the attention on “too” and “tired”, which is not the semantic correlation we expect. In contrast, AMPLIFY made a more accurate choice by putting the attention on “animal” and “it”. This is consistent with our understanding that “it” should refer to “animal”.

In the semantic relationship diagram of the word “it” shown in Fig. 8, because the sentence initially defines “film”, expresses dissatisfaction with a certain type of movie, and finally expresses a positive attitude towards “film”, it can be inferred that the “it” appearing in the sentence refers to “film” based on the consistency of semantic logic and the way the sentence expresses emotions. However, common pre-trained text models such as BERT split the keyword “cartoonish” into two lower-granularity tokens “cartoon” and “#ish” in order to solve the out-of-vocabulary problem, which obviously led EmbedMix to not consider that “cartoonish” is a complete adjective used to describe a certain movie genre. On the other hand, AMPLIFY not only focuses on “film”, but also extends attention to the grammar dependencies between subjects and predicates such as “#ish”, and “provides”.

For the sake of fairness, both mixup methods used in this section to compare respectively copy the outputs of the hidden layers at the same position, and perform mixup after shuffling. Other settings are consistent with those in the ‘Experimental Settings’ section. In addition, all attention visualization patterns in this section are from Vig (2019) and Wang, Turko & Chau (2021).

Conclusion and Future Work

This article proposes a novel and simple hidden layer mixup method called AMPLIFY, which solves the limitations of the standard mixup method’s sensitivity to noise information and out-of-domain features. By performing mixup operations on all MHA layers of pre-trained language models based on Transformer architecture and using mild random perturbation terms to augment the explanatory feature sequences of each attention mechanism output, AMPLIFY suppresses the effects of noise information and out-of-domain features on the mixed results. Compared with standard data augmentation strategies, AMPLIFY can better control the fluctuations in model performance gains. Compared with traditional mixup methods, AMPLIFY has better robustness and generalization. The experimental results show that our proposed method has practical significance for exploring the performance potential of models in different NLP tasks. In addition, the AMPLIFY method has high computational efficiency, avoiding the partial resource overhead required by other mixup methods, reducing the overall cost of the algorithm, and making it more engineering-oriented. The experimental results also show that using mixup in different MHA layers is a more effective choice depending on the features of different datasets. However, constructing a mixup method that can dynamically adjust the structure and application location for different datasets is a very difficult task. More generally, research on the learning rules of the BERT model also shows that the semantic features and grammar-related information learned by the model in different network layers are different, and the benefits brought by them are also different. Therefore, it is recommended to perform appropriate mixup on different network layers to obtain more significant overall performance gains, which is consistent with the idea of AMPLIFY.

For future work, we believe that there is still considerable exploration space in how to combine mixup operations with various attention mechanisms. In addition, extending and optimizing existing mixup methods is also a potential opportunity. For example, our next research direction is to apply AMPLIFY to models in other fields outside of NLP classification tasks, such as ViT (Vision Transformer) or Contrastive Language-Image Pre-training (CLIP), to evaluate its applicability and effectiveness. Additionally, we plan to combine AMPLIFY with other data augmentation techniques (such as Test-Time Augmentation) to further improve the performance of pre-trained language models.

Supplemental Information