Joint coordinate attention mechanism and instance normalization for COVID online comments text classification

Overall plan

Let a text sequence with m words be represented as $x=\left[x_1,\dots ,x_m\right]$, and let the set of labels be $\wp =\left\{l_1,\dots ,l_c\right\}$, where c denotes the total number of categories (such as the binary classification used in this study, c = 2). The labels are divided into negative and positive categories, which prepare the model for learning and final validation. Note that the text sequence x in the document contains m words, and the label sequence l in document contains ℘ labels. The mapping process is shown in the following equation. The label are associated with the note text denoting z_text, and the label with the note text denoting z_label.

${\displaystyle z_{text},z_{label}=f_{enc}\left(x,l\right),}$ where f_enc represents the process of encoding a mapping function. After entering the two representations of z_text and z_label obtained using above formula, we use the decoder to generate the probability sequence $\hat{y}$ as follows: ${\displaystyle \hat{y}=f_{dec}\left(z_{text},z_{label}\right),}$ where f_dec represents the process of decoding the mapping function. Using the above equation, the decoder can use the mutual representation of the text and labels to make a final prediction.

INTLCE

In this section, we discuss the INTLCE module in detail. INTLCE encodes text and label sequences into mutually participating labels and text representations. Specifically, INTLCE is divided into the projection, text label coordinates attention and fusion encoder layers.

We used the bidirectional long short term memory (BiLSTM) (Graves & Schmidhuber, 2005) approach to effectively encode words and labels and enhance our understanding of input sequences. In addition, this approach helps identify correlations between labels, which further improves the accuracy of the encoding process.

Given a sequence of words and labels x ∈ R^m and l ∈ R^c, we start by mapping words and labels separately to the word-embedding layer x_emb ∈ R^m×d_emb and label-embedding layer L_emb ∈ R^c×d_emb respectively, where d_emb denotes the embedding dimension.

For each sentence, we used pre-trained GloVe word embedding (Kamyab, Liu & Adjeisah, 2021; Ibrahim et al., 2021). This is an unsupervised word-vector representation technique. The resulting representation depicts the linear substructure in the word vector space by training the aggregated global word-word co-occurrence statistics in the corpus.

For each label, a random initialization was used for embedding. To improve computational efficiency, we first used an independent linear projection layer. Suppose x_enb and L_emb are projected separately into a more compact, smaller-dimensional embedding, where X_proj ∈ R^m×d, L_proj ∈ R^c×d, d < d_emb, d is a hidden dimension.

Thereafter, we used the BiLSTM method for word embedding X_proj ∈ R^m×d and label embedding L_proj ∈ R^c×d in the projection layer. The calculation equations are expressed as follows:

${\displaystyle \begin{array}{c}{X}_{enc}=BLM\left(X_{proj}\right),\hfill \\ L_{enc}=BLM\left(L_{proj}\right),\hfill \end{array}}$ where BLM is short for BiLSTM, X_enc denotes text encoding and L_enc is the label coding. In our implementation, we incorporated weight sharing into the BiLSTM. This means that the same weights are used for both the forward and backward directions of the network, which can help reduce the number of parameters and improve efficiency.

Furthermore, the study introduces a novel normalization technique, termed IN, which represents an innovative approach to the standardization of data within the experimental framework. This method enhances the analytical rigor by ensuring consistency in data treatment, thereby improving the reliability and interpretability of the results. IN is independent of the channel or batch size and ensures the independence of each text instance, which enhances the performance of the model. Another optimization approach involves replacing the original self-attention mechanism activation function with the GELU functions. Random regularization was added to make the model more consistent with the cognitive processes.

First, the key steps in optimizing the coordinate attention mechanism focus on the scaled dot product attention, which is expressed as follows:

${\displaystyle Attention\left(Q,K,V\right)=Softmax\left(\frac{Q{K}^T}{\sqrt{d_h}}\right)V,}$ where Q ∈ R^q×d_k, K ∈ R^k×d_k, and V ∈ R^v×d_v. The multi-head attention equation is expressed as follows: ${\displaystyle MHead\left(Q,K,V\right)=Softmax\left(\left[H_1;...;H_p\right]\right)W^0,}$ where $H_i=Attention\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right)$, and the projection parameters are $W_i^Q\in R^{d_k\times d_p}$, $W_i^K\in R^{d_k\times d_p}$, $W_i^V\in R^{d_v\times d_p}$, and $W_i^O\in R^{p{d}_p\times d_h}$. We used d_k = d_v = d_p and d_p = d_h/p as the dimensions of each head, where d represents the dimension of the interval model, p represents the number of heads, and $\left[\cdot \right]$ represents the join operation.

Next, because the traditional self-attention mechanism (Vaswani et al., 2017) only considers text modes, the matrices Q, K, and V represent text encodings. The text code X_enc and label code L_enc were simultaneously input into multiple attention modules, and the self-attention module was converted into a coordinate attention module.

${\displaystyle \begin{array}{c}{X}_{att}=MHea{d}_X\left(X_{enc},L_{enc},L_{enc}\right),\hfill \\ L_{att}=MHea{d}_L\left(L_{enc},X_{enc},X_{enc}\right),\hfill \end{array}}$ where X_att ∈ R^m×d_h and L_att ∈ R^c×d_h denote the text representations of the label participation and text participation, respectively. The term d_h represents the hidden dimensions of the coordinate attention layer.

Furthermore, after IN, the residual connection and feedforward network (FN) obtain text and label fusion, encoding X_fu ∈ R^m×d and L_fu ∈ R^c×d.

${\displaystyle \begin{array}{c}{X}_{fu}=I{N}_X\left(F{N}_X\left(X_{att}\right)+X_{enc}\right),\hfill \\ L_{fu}=I{N}_L\left(F{N}_L\left(L_{att}\right)+L_{enc}\right).\hfill \end{array}}$ The FN maps the input to a higher dimension d, enabling mutual engagement between the text and label.

This study adopts a case normalization method that differs from that reported in a previous study (Liu et al., 2022) in the processing of text features. This method not only avoids dependence on a small range of neurons but also maintains the independence of text instances and accelerates the convergence speed of the model regardless of the channel or batch size. The normalization formula of IN is expressed in equations as follows:

${\displaystyle y_{tijk}=\frac{x_{tijk}-{\mu }_{ti}}{\sqrt{{\sigma }_{ti}^2+\in }},}$ ${\displaystyle {\mu }_{ti}=\frac{1}{HW}\sum _{l=1}^W\sum _{m=1}^H{x}_{tilm},}$ ${\displaystyle {\sigma }_{ti}^2=\frac{1}{HW}\sum _{l=1}^W\sum _{m=1}^H{\left(x_{tilm}-m{u}_{ti}\right)}^2.}$ For the selection of the activation functions, GELU (Lee, 2023) were selected. GELU is a high-performance neural network activation function that has been successfully applied to the BERT model (Hendricks & Gimpel, 2016). GELU functions exhibit excellent generalization and stable optimization abilities, which can improve model performance and reduce the difficulty and time cost of model training. The mathematical expression of the GELU is reproduced as follows:

${\displaystyle GELUs\left(x\right)=xP\left(X\le x\right)=x\Phi \left(x\right),}$ where Φ(x) denotes the probability function of a normal distribution.

To fully exploit the information encoded by the text engaged by the label and the relevance encoded by the label engaged by the text. In this study, a fusion encoder layer was introduced into the model and two mutually independent BiLSTM layers were constructed to propagate the fused text and label information. One BiLSTM layer was used to generate the ultimate text representation X_fin by combining the text encoding X_fu as follows:

${\displaystyle X_{fin}=BL{M}_X\left(X_{fu}\right),}$ where X_fin ∈ R^m×d. The subsequent decoding process is preserved in the hidden state h ∈ R^d×1 and cell state c ∈ R^d×1 of BLM_X. These are utilized to initialize the hidden and cell states of the LSTM decoder, which assists in generating a logical output from the input sequence.

Another BiLSTM encoder fuses the label encoding to produce a sequence of the label L_fin ∈ R^c×d as follows:

${\displaystyle L_{fin}=BL{M}_L\left(L_{fu}\right).}$

ALD

For the final component of the model, we employed an ALD architecture. ALD consists of two steps for each time step. First, using the LSTM decoder, the hidden, cell, and loop context states were acquired in the first step of ALD decoding. Second, the probability of each class was determined using this component, which handles the classification problem simultaneously without modifying the model (Yang et al., 2018).

In the first step, we used LSTM cell attention as the benchmark technique for implementing the LSTM decoder. During training, we initially locate the label embeddings e_t−1 of the $\left(t-1\right)$ decoding step in the true label and the predicted label embedding. We will use this alignment during the prediction process. Thereafter, the LSTM cell takes as input e_t−1, the recurrent context state r_t−1, the hidden layer state h_t−1, and the previous time step of the cell state c_t−1. It outputs the hidden state h_t and cell state c_t of the current time step t, as expressed in the following equation.

${\displaystyle h_t,c_t=LSTMCell\left(\left[e_{t-1};r_{t-1}\right],h_{t-1},c_{t-1}\right),}$ where h and c denote the coding processes to initialize h₀ and c₀, respectively.

And then initialize e₀ and r₀. Once we obtained the hidden state h_t, we calculated the result of a_t when the step size is t. The adaptive classifier receives the hidden state h_t for subsequent processing.

${\displaystyle a_t=Softmax\left(X_{fin}{W}_1{h}_t\right),}$ where W₁ denotes a trainable matrix, and r_t is the state of the context at time step t, as expressed in the following equation.

$r_t=\mathrm{Tanh}\left(W_2\left[X_{fin}^T{a}_t;h_t\right]\right)$,

where W₂ ∈ R^d×2d represents a trainable matrix.

We used an adaptive classifier in the final classification phase to utilize the label representation of information-text participation. In comparison with most existing methods, the adaptive classifier can directly output the probability of each class by focusing on the label representation of text participation.

Label probability prediction

Given the hidden state h_t of the step size t, this classifier considers the final label of the code L_fin and the hidden state h_t as inputs to obtain the probability ${\hat{y}}_t$ of the time-step t.

${\displaystyle {\hat{y}}_t=Softmax\left(L_{fin}{W}_3{h}_t\right),}$ where W₃ represents a trainable matrix and ${\hat{y}}_t$ is the output probability in each class. Therefore, to simplify the classification process, we integrated label representation into the text-attention mechanism.

After obtaining the probabilities for all the categories, we optimized our model by computing the objective function.

Dataset

An experimental dataset called the COVID dataset was constructed by crawling COVID online reviews from December 2019 to March 2022 using Python Creeper technology. The constructed COVID dataset totaled 100,000 records. In the COVID dataset, Label 0 was designated as a negative sample, whereas Label 1 was considered a positive sample. The COVID dataset for the number of positive and negative sample distributions is shown in Table 1.

We also used Baidu’s AI open platform (https://ai.baidu.com) to conduct a series of early stages. First, data capture. Large amounts of data are collected from various online web pages. By adjusting parameters such as keywords, page numbers, results per page and domain Settings, Baidu ensures that comprehensive data is collected according to specific needs. Second, handle paging. To crawl multiple pages of search results, the crawl script systematically increases page parameters to cover the required number of pages. Finally, preliminary data cleaning. This includes deleting duplicate data and detecting errors.

To better analyze the data, further data cleansing was done, such as removing Spaces, URL address information, and hashtags. Doing so will prepare you for the subsequent preprocessing phase.

The purpose of the text preprocessing stage is to carry out a series of processing and conversion of the cleaned text in order to facilitate the subsequent development feature extraction and model training. Data preprocessing is divided into two steps. The first step is text segmentation. Using Jieba word segmentation method deals with the review data, which is of great help to some new words and words not included in the dictionary. Step two, build stop word list. This article refers to the common stop word list after adding a custom stop word, in order to better handle comments on the data. Each preprocessed COVID dataset included the comments and the corresponding sentiment label. The data is shown in part for example, see Table 2.

Evaluation indicators

We employed the micro-averaged F1 score (Mi-F1) to evaluate the performance of our model. The Mi-F1 is a common metric used in evaluating classification models, especially in scenarios where imbalanced class distributions are present. The micro-averaging method treats the contributions of all categories equally, making it suitable for datasets with imbalanced sample distributions, as it calculates overall performance by aggregating the results across all categories. In practical applications, the Mi-F1 pays more attention to categories with a large number of instances, which is particularly important in applications such as text classification. The calculation method of the Mi-F1 is straightforward, easy to understand, and implement, making it a widely adopted tool among researchers and practitioners.

For category i, true positives are denoted as TP_i, false positives as FP_i, and false negatives as FN_i. The calculation formula is expressed as follows:

${\displaystyle Mi-F1=\frac{{\sum }_{i=1}^{c}2T{P}_i}{{\sum }_{i=1}^{c}2T{P}_i+F{P}_i+F{N}_i}.}$

Model parameter setting

In this section, we concatenated all the labels in a randomly selected order. The resulting label sequence was identical for all samples in the dataset and included every label in the label set. Using the same label sequence for all the samples, the model learned to recognize and classify each label consistently across the entire dataset. This can enhance the performance and provide more reliable results. The experimental settings used in this study are listed in Table 3.

Experimental result

Large pre-training models, such as BERT (Devlin et al., 2018), XLNet (Yang et al., 2019), ERNIE 3.0 (Sun et al., 2021), and NABoE (Yamada & Shindo, 2019), have recently gained popularity and have also acquired powerful language representation capabilities using large-scale unsupervised corpora. Some of these models outperformed our model; however, the number of parameters in these pre-trained models was large, making them less computationally efficient. For instance, the ERNIE 3.0 pre-training model exhibited the best performance but with billions of parameters. The BERT pre-training model had relatively few parameters; however, the number of parameters reached 340 million. Figure 2 shows the performance of the model in this study compared with four large pre-training models. We can clearly observe that the proposed model exhibits certain shortcomings in terms of performance. The pre-training model with the lowest number of parameters had approximately 68 times the number of parameters of the proposed model. In real-world scenarios, where time and space constraints are critical, the sheer number of parameters required by pre-training models can pose a significant disadvantage. These models often require extensive computational resources and are difficult to deploy in practical applications with limited resources. Therefore, it is important to balance model performance with practical constraints when selecting a mode for a given task.

We conducted an experimental evaluation of the proposed model in comparison with several baseline models on the COVID dataset, and compares the time complexity of each model, the performance of the prediction algorithm in large-scale data and resource use efficiency has a larger advantage. Table 4 summarizes the results of this comparison.

The DocBERT model (Adhikari et al., 2020) demonstrates superior performance in document classification by fine-tuning the BERT model.

The LSTM model (Chen, Tseng & Wang, 2021) possesses a sophisticated architecture that effectively manages long-term dependencies. Through the integrated function of its input, output, and forget gates, it dynamically controls the retention and omission of information within an LSTM unit at any given moment.

The Very Deep Convolutional Networks for Text Classification (VDCNN) (Conneau et al., 2016) employ minimal convolutional and pooling operations for character-based classification, enabling the model to incorporate 29 convolutional layers, thus facilitating deep-text classification.

The HTTN model (Xiao et al., 2021) introduces a novel head-to-tail network that capitalizes on the relationship between the head and tail labels to facilitate the transfer of meta-knowledge from data-rich tail labels to those with less data.

The LEAM model (Wang et al., 2018) innovatively embeds each label within the same vector space as the word vectors, utilizing an attention mechanism to gauge the compatibility between the text sequence and the label embeddings.

The LSAN model (Xiao et al., 2019) represents a label-specific attention network optimized for multi-label text classification tasks. It effectively exploits the semantic relationships between labels and words to enhance classification accuracy.

The CNLE model (Liu et al., 2022) deeply integrates sequence information from both labels and texts. This integration captures comprehensive representations by amalgamating input from both the text and labels, thereby enhancing the precision and comprehensiveness of the classification.

As shown in Table 4, label embedding-based methods (HTTN, LEAM, and LSAN) generally surpass text representation methods (DocBERT, LSTM, VDCNN), underscoring the effectiveness of label-oriented strategies in improving model performance.

The CNLE model utilizes labeled representations in conjunction with textual interactions, allowing for the labeling of text engagements and thereby enriching the data. This improvement is achieved through the IN method, which independently normalizes each sample, reducing variance across different input data distributions. This normalization significantly enhances the model’s ability to generalize across diverse datasets. Additionally, IN accelerates training convergence and improves stability by mitigating the Internal Covariate Shift (ICS)—the impact of input data distribution changes on model training. Therefore, the model described in this manuscript employs the IN method to enhance its generalization capabilities and stabilize the training process. Moreover, by integrating sequence information from both text and labels, IN helps the CNLE model process long textual data more effectively and capture contextual subtleties.

This study introduces the Gaussian Error Linear Unit (GELU) as the activation function in the proposed model to facilitate nonlinear improvements. The GELU function, by allowing the passage of small negative inputs, enables richer nonlinear transformations and the formation of more complex decision boundaries compared to the traditional ReLU. This is particularly beneficial in text categorization, where identifying subtle textual nuances is crucial. Within the CNLE framework, the GELU function enhances the interaction between text and label data, thus increasing the expressive power of the model’s features. This leads to more accurate text classification, especially in cases involving ambiguous or complex classification criteria.

In summary, the integration of the CNLE-based IN method and the GELU function significantly boosts the effectiveness of the text categorization model discussed in this paper. This approach is particularly effective in managing diverse and extensive textual data, markedly improving both the accuracy of classifications and the robustness of the model.

Ablation studies

Two ablation studies are conducted to test the validity of the proposed model. The results are shown in Fig. 3.

A none-labeled model was used in the first ablation study, where only a sequence of texts was used as the input to the model. Most of the existing models similarly approach this method. In the second ablation study, both label embedding and label text attention components were retained in our approach (referred to as label-to-text). However, text label attention has been eliminated in our approach, ensuring that no text is incorporated in the label embedding. Our approach incorporates a text-label coordinate attention architecture, and the ablation results demonstrate that using a label-attention text representation and text-attention label representation can effectively enhance classification accuracy. This finding highlights the importance of considering both text and label components and suggests that incorporating a coordinated approach between the two can enhance performance.

Parameter analysis

In this section, we experimentally explored the effects of the hyperparameters on the overall performance of the proposed model. The hyperparameters involved in the experiment primarily included the number of attention heads in the multi-head attention layer and the size of the hidden layer in the coordinate attention mechanism.

The number of attention heads in the multi-head attention mechanism is an important hyperparameter, which determines the number of positions on which the model can focus. Increasing the number of attention heads can improve the expression and learning ability of the model such that the model can learn the relationship between different positions and features simultaneously to better capture the information of the input sequence. However, too many attention heads can lead to overfitting or performance degradation; therefore, an appropriate selection is required. The experimental results are listed in Table 5.

The data presented in Table 5 clearly illustrates that the performance of the proposed model is suboptimal when the attention head count is set to one. This diminished efficacy is attributable to the reduction of the multi-head attention mechanism to a single original attention model, which compromises the precise allocation of weight information across different positions. As the number of attention heads increases, there is a corresponding improvement in model performance, reaching an optimal state when the count is three. However, an escalation in the number of attention heads beyond this point results in a deterioration of performance, indicative of model overfitting. This phenomenon suggests that a higher number of attention heads may lead to excessive model complexity, which negatively impacts generalization.

Thereafter, we evaluated the effect of the hidden size of the coordinate attention mechanism, ranging from 100 to 300, on the performance of the model. The experimental results are listed in Table 6.

As summarized in Table 6, the model performance gradually decreased as the size of the hidden layer in the coordinate attention mechanism increased. The decrease in performance is attributed to the increase in the dimension of word representation, which causes an increase in the training difficulty of the model, resulting in underfitting.

Case study

In this section, experiments are conducted on two datasets, Yelp Polarity Reviews and Amazon Polarity Reviews, to further evaluate the model.

The Yelp Polarity Reviews dataset is extensively utilized in natural language processing and machine learning research, particularly for sentiment analysis. This dataset includes restaurant reviews from Yelp users, each annotated with either a positive or negative label, representing positive or negative sentiment respectively. It comprises over 500,000 reviews, evenly distributed between positive and negative sentiments.

Similarly, the Amazon Polarity Reviews dataset is frequently employed for sentiment analysis tasks. This dataset consists of product reviews from Amazon users, each labeled with positive or negative sentiment. It contains millions of reviews, providing a substantial resource for analyzing customer feedback and sentiment trends.

The experimental results of our proposed model for text classification on these two datasets are presented in Table 7.

Due to the lengthy training times required by deep learning models, the outcomes derived from various hyperparameters across different datasets will vary. Table 7 illustrates that the results achieved using the hyperparameters mentioned in the previous subsection are suboptimal for both datasets. To improve these results, extensive experimentation is necessary to further refine the model’s hyperparameters. However, this paper does not include extensive hyperparameter tuning experiments for these datasets.