Online teaching emotion analysis based on GRU and nonlinear transformer algorithm

Sentiment feature extraction module

In multimodal sentiment analysis methods, data alignment plays a crucial role in scaling and aligning modal features. However, this alignment process often results in the loss of vital feature information. To address this issue, we introduce the lightweight attention aggregation module, primarily comprising a bi-directional gated recurrent unit (Bi-GRU) neural network. This module takes in speech and video data, capturing feature information and offering potential alignment cues for subsequent data processing. Its key advantage lies in its efficient extraction of information from low-resource modal data, eliminating the need for extensive data alignment.

Furthermore, the lightweight attention aggregation module contributes low-resource modal data features to the overall model, allowing for fine-tuning the accuracy of the final output. Consequently, inputting low-resource modal data into this module does not necessitate overly complex processing, resulting in enhanced efficiency in feature extraction. Figure 2 provides an illustration of the modal sentiment feature extraction module’s structure.

The primary objective of this module is to extract feature information from resource-limited modal data. In the figure, the input data consists of unprocessed images and raw speech data. The dashed data within the module signifies parameter updates, while the resultant output data comprises the respective 1D feature tensors for speech and images. The module encompasses a data processing layer and a feature generation layer. The data processing layer includes a convolutional layer, layer normalization, and a data mapping section, while the remaining data constitutes the feature generation layer.

The feature processing layer prepares the data for feature extraction in the lower layer. The following calculation process uses image data I and speech data A as examples. The data A passes through a convolutional layer with parameters from the Wav2Vec model. In contrast, the data I passes through a convolutional layer with only one parameter randomly generated because audio data needs to be transformed into waveform features before the model can process it. The convolutional squared data method yields the tensor $Z_A$, with the operator expression shown in Eq. (1), where $T_A$ represents the total number of speech samples and $d_A$ is the tensor dimension of a single speech sample. (1) $$Z_A=CN{N}_{\theta }(A)=\{Z_1,Z_2,...,Z_{T_A};Z\in R^{d_A}\}$$

The data normalization layer offers the benefit of enhancing the convergence speed of the model while remaining unaffected by the sample size, thereby rendering it suitable for a broader array of scenarios. Data mapping, accomplished through linear regression, is an efficient approach for preprocessing data by partitioning distinct types using a hyperplane. This enables rapid model convergence. The calculation process is illustrated in Eq. (2). (2) $$P_A=Projection(L_A)=W_{\mathrm{\Phi }}^T\times L_A+b_{\mathrm{\Phi }}$$where $W_{\mathrm{\Phi }}$ is used to determine the direction of the hyperplane, the bias $b_{\mathrm{\Phi }}$ represents the offset of the hyperplane and $P_A$ is the refined feature data after data mapping.

The feature generation layer leverages the effectiveness of the GRU model to extract features efficiently. It employs the attentional feature enhancement method and the one-dimensional information aggregation technique to extract crucial feature information from low-resource modal data. With the attention weights determined, the model computes a weighted sum of the features extracted by the GRU. This means that the features that are deemed more important (according to the attention weights) contribute more to the final representation, while less important features have a smaller influence. Generating a one-dimensional information aggregation matrix enhances the significant information of the low-resource modal data, thereby mitigating the risk of information loss.

The feature generation layer receives the mapped data and performs several rounds of operations with the GRU to generate the coarse feature information matrix $M_{\mathrm{\Phi }}$, as shown in Eq. (3) and the vector representation of $M_{\mathrm{\Phi }}$ are shown in Eq. (4), (3) $$M_{\mathrm{\Phi }}=GR{U}_{{\theta }_{GRU-A}}(P_A)$$ (4) $$M_{\mathrm{\Phi }}=\{{\varphi }_1,{\varphi }_2,...,{\varphi }_{d_A}\},i\in [1,d_A]$$where is the random initialization parameter of the GRU model and is the vector corresponding to each feature information in the coarse feature information matrix.

Cross-modal transformer module

The proposed cross-modal Transformer employs a two-by-two multimodal fusion approach. It utilizes text data as the target matrix and maps speech and video to text, resulting in two cross-modal mapping tensors. These mapping tensors are fused using the same approach to generate a tri-modal mapping tensor. In the multilayer training structure, the cross-modal Transformers employ a feed-forward fusion process to merge the multimodal sequences. Each modality (e.g., images and text) is usually processed separately, and the data from each modality is represented as tensors. These tensors could represent features extracted from the data. Once the mapping is learned, the tensors from different modalities can be combined or fused in various ways. The specific fusion method depends on the problem at hand. For example, you might concatenate the tensors, perform element-wise operations, or use attention mechanisms to give different modalities different weights.

After fusion, the combined tensor is typically passed through additional layers of a neural network or other processing steps to make predictions, classify objects, or perform some other task.Each cross-modal Transformer learns the attention between two modal features while leveraging the underlying features of another modality to enhance the target information repeatedly. This enables effective multimodal fusion and efficient utilization of multimodal feature information. Figure 3 illustrates the structure of the cross-modal Transformer module, with the input data comprising text, speech, and image data, respectively. The output data consists of the cross-modal mapping tensor, and the arrows in the figure indicate the data transfer.

The module commences with a data preprocessing layer, employing a convolutional reshape data technique and additional positional coding. Convolutional reshaping involves utilizing text modalities as the target modal matrix and low-resource modalities as the mapping modal matrix. It also establishes a fusion relationship by leveraging a multiheaded attention mechanism to capture feature information. The additional position encoding method incorporates position encoding to incorporate information regarding the relative or absolute position of each element within the sequence. This is represented as demonstrated in Eqs. (5) and (6). (5) $$P{E}_{(pos,2i)}=\sin (pos/{\mathrm{10,000}}^{2i/d_{mod\mathit{e}\mathit{l}}})$$ (6) $$P{E}_{(pos,2i)}=\cos (pos/{\mathrm{10,000}}^{2i/d_{mode\mathit{e}\mathit{l}}})$$where $pos$ is the position of a token in the sequence, $i$ represents the currently used formula selected by the parity of pos, and $d_{mode\mathit{e}\mathit{l}}$ represents the embedding dimension.

Subsequently, a cross-modal multiheaded attention layer is employed to facilitate the establishment of long-range dependencies. Each element is projected into a distinct space through the dot product of Query, Key, and Value, enabling the extraction of diverse feature information. The multilayer training method is an optimization technique for the model, enabling the fitting of complex functions while effectively reducing the number of network parameters and significantly improving model learning efficiency. Feed-forward neural networks, with their nonlinear learning capabilities and the ability to discover nonlinear features throughout the network, facilitate the efficient extraction of multiheaded information and optimization of feature information.

Since $q$ and $k$ are independently distributed, their expectations are both 0 and their variances are both 1, as shown in Eqs. (7) to (9), where the value of $d_k$ is the total number of elements. (7) $$E(Z)=E(X_q\cdot X_k)=E(X_q)E(X_k)=0$$ (8) $$D(Z)=D(X_q\cdot X_k)=D(X_q)D(X_k)=1$$ (9) $$D\left(\sum _{i=1}^{d_k}{Z}_i\right)=\sum _{i=1}^{d_k}D(Z_i^{})=d_k$$where E(Z) is the expectation (mean) of the random variable Z, and D(Z) is the variance of the random variable Z.

Emotional level output module

The effective polarity output module incorporates an additional cross-modal Transformer module, a Concat layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The structure of this module is illustrated in Fig. 4. Once the output from the cross-modal Transformer is received, the data proceeds to the emotional polarity output module. This module encompasses a Concat layer, a convolutional layer, a pooling layer, a fully-connected layer, and an output layer.

The data after the cross-modal Transformer is the information superimposed on the three modes $C{M}_{A->I,I->L}(L,A,V)$, which is superimposed on the trimodal information. In the Concat layer, $C_A$ and $C_I$ are tensor stitched with $C{M}_{A->I,I->L}(L,A,V)$, convolved, pooled and then output as a floating point number of samples in the fully connected layer.

The final output of the model is a floating point number between [−3, 3]. The seven integers in the interval represent the sentiment polarity predicted by the model, which also allows the sentiment level of the model output to be judged by its positivity or negativity, meaning that the method in this article can perform both dichotomous and multiclassification tasks. The model output data will be compared with the data labels to evaluate the model performance. Feature fusion methods are an essential approach in the field of pattern recognition. Feature fusion methods can combine the features of multiple modalities to achieve the advantages of various features to complement each other. They can obtain recognition results with more robustness and accuracy.

Evaluation indicators

The confusion matrix, referred to as the likelihood matrix or the error matrix, is a valuable visualization tool commonly employed in supervised learning. It encompasses four distinct states: true positive (TP) denotes positive samples correctly predicted as positive, false negative (FN) represents positive samples mistakenly predicted as negative, false positive (FP) signifies negative samples erroneously predicted as positive, and true negative (TN) corresponds to negative samples correctly predicted as negative. From these states, metrics such as recall and accuracy can be derived. Furthermore, the evaluation index of choice, Acc, assesses the overall proportion of accurate classifications. The F1 score captures the comprehensive performance of multi-classification tasks, while MAE quantifies the proportion of misidentifications. Lastly, Corr gauges the level of correlation between two components of a two-dimensional random variable. Corresponding formulas are as follows:

Acc = (Number of Correct Predictions)/(Total Number of Predictions).

Precision = (TP)/(TP + FP).

Recall = (TP)/(TP + FN).

F1 Score = 2 * (Precision * Recall)/(Precision + Recall).

MAE = Σ |Actual − Predicted|/n (summing over all data points).

Coor = Σ [(X − $\overline{\text{X}}$) * (Y − $\overline{\text{Y}}$)]/[√Σ(X − $\overline{\text{X}}$)² * Σ(Y − $\overline{\text{Y}}$)²].

where X and Y are the variables of interest, $\overline{\text{X}}$ and $\overline{\text{Y}}$ are their respective means, and Σ denotes summation over all data points.

Ablation experiments

To evaluate the efficacy of the proposed modal feature extraction module (MFEM), an ablation experiment is conducted, denoted as MFEM hereafter. In this experiment, common feature extraction modules such as CNN, RNN, and LSTM are selected as replacements for MFEM. Moreover, to assess the effectiveness of the proposed Trans-modal Transformer module, the following modules are chosen for ablation experiments: RAVEN, BBFN, and Tri-TransModality. Additionally, for comparison, the Attention Gating structure (AG), LSTM memory module (MM), and Attention Fusion Module (AFM) are selected. These modules aim to verify the effectiveness of the cross-modal multiheaded attention module. The outcomes of the experiments are presented in Tables 4– 6.

Based on the data presented in the table, it is evident that all three modules proposed in this article outperform their counterparts across six evaluation metrics. This superiority can be attributed to the design of the trans-modal feature extraction, which exhibits enhanced feature fusion capabilities compared to traditional methods. Notably, the performance of the proposed cross-modal Transformer surpasses that of the second-best performing module, Tri-TransModality, by margins of 0.0444, 0.0344, 0.0377, 0.0453, 0.0015, and 0.0015 across the evaluation metrics. This significant difference underscores the effectiveness of the cross-modal fusion design in sentiment recognition. Furthermore, the proposed cross-modal multiheaded attention module outperforms the second-best performing AFM module by margins of 0.0211, 0.0164, 0.0278, 0.0156, 0.0020, and 0.0008 across the six metrics. This outcome suggests that our attention mechanism approach, incorporating the cross-modal structure, enables the model to focus on the most salient aspects of each modality better.

Comparative experiments

Several comparable approaches were selected for evaluation to ascertain the effectiveness of the proposed cross-modal student sentiment analysis model. These approaches are as follows:

CTC + RAVEN: This approach utilizes parallel LSTM models to process multimodal data independently. It incorporates an attention-gating module and employs the word vector alignment technique to align the multimodal data.

BBFN: This approach employs LSTM memory and cyclic self-connection modules to model the multimodal data. Gating modules are utilized to enhance model training.

CubeMLP: This approach employs separate LSTM models to process sound and image data. The features of the text modal data are abstracted using a transformer encoder. Finally, an MLP fusion network is employed to fuse the multimodal information.

CM-BERT: This approach fine-tunes the BERT model by leveraging the interaction between text and audio. A multimodal attention mechanism dynamically adjusts words’ weight using text and audio information.

The results of the comparative experiments are presented in Table 7.

The tabular data provided in this study unequivocally demonstrates the consistent superiority of the method expounded within this research article when juxtaposed with analogous approaches, as elucidated by a comprehensive evaluation involving six distinct metrics. These empirical findings underscore the remarkable efficacy of the proposed method in the realm of information extraction from multimodal data, coupled with the precision it offers in the domain of sentiment analysis, through the strategic amalgamation of multimodal techniques and Transformer-based architectures. The intrinsic aptitude of the method to proficiently dissect and analyze information stemming from diverse modalities is undeniably manifest.

Furthermore, the elucidation of these findings is supplemented by Fig. 5, which illustrates the confusion matrix stemming from the application of the proposed method to a self-constructed dataset. Notably, this visual representation unveils a sparse distribution of misclassified instances, scattered sporadically across the spectrum of five distinct sentiment categories. Significantly, the preponderance of these misclassifications gravitates toward the vicinity of the diagonal axis, suggestive of the method’s resilience and robust generalization capacity. It is worth emphasizing that such misclassifications are few and specific, indicating that the proposed method’s susceptibility to erroneous categorization is confined to isolated and atypical instances within the dataset.