Analysis of the fusion of multimodal sentiment perception and physiological signals in Chinese-English cross-cultural communication: Transformer approach incorporating self-attention enhancement

Sentiment analysis using single modality

Sentiment analysis (Wankhade, Rao & Kulkarni, 2022) involves inferring hidden emotions from data by categorizing them into predefined artificial categories. Traditional sentiment analysis methods, such as text based on sentiment lexicons (Subhashini et al., 2021), rely on calculating sentiment scores by checking whether the text contains specific lexicons from a pre-established dictionary or the polarity of phrases within the text. These lexicons are typically limited to a few thousand words due to the manual creation of sentiment dictionary libraries (Widmann & Wich, 2023). Subhashini et al. (2021) analyzed customer reviews by counting the occurrences of positive and negative words, with the predominant polarity determining the sentiment of the comment. Yadav & Vishwakarma (2020) employed the bag-of-words model to extract textual sentiment features and assessed the effectiveness to sentiment analysis (AlBadani, Shi & Dong, 2022). Yu et al. (2020) constructed a Bi-LSTM model with attention enhancement for speech emotion analysis, achieving high accuracy. Compared to text and speech, facial expressions are more adept at conveying emotions during interpersonal communication, leading to significant research advancements in this domain. Abdat, Maaoui & Pruski (2021) proposed a face key point detection model, which measures changes in facial expressions through 21 distances on the human face. Pei et al. (2025b, 2025a) utilized a CNN-LSTM network to extract spatial features of expressions between frames, applying these features for emotion classification. In recent years, researchers have begun exploring multimodal sentiment analysis methods, integrating multiple information sources—such as text, speech, and video—to obtain a more comprehensive understanding of sentiment.

Sentiment analysis based on multimodal data

Early research in the domain of multimodal sentiment analysis primarily focused on static graphical and textual federated data, with the objective of establishing a mapping relationship between textual-visual features and sentiment semantics to enable sentiment description in social media’s graphical-textual fusion data (Li et al., 2019). With the advent of deep learning, its strong feature representation capabilities have made it the dominant approach in multimodal sentiment analysis. Zadeh et al. (2017) employed a three-fold Cartesian product to fuse multiple unimodal information instead of using simple tensor concatenation. Liu et al. (2018) introduced a multimodal fusion method to enhance the efficiency of multimodal fusion. Some researchers have approached the problem from the perspective of decision fusion. Akhtar et al. (2019) and Ye et al. (2024) decomposed the multimodal sentiment analysis task into several sub-tasks, conducting joint training and performing comprehensive sentiment analysis based on the predictions of each task. Hazarika et al. (2018) used an interactive conversational memory network for multimodal sentiment detection, where global memory is used to capture contextual summaries. Zhang et al. (2019) introduced a quantum-inspired interaction network, combining quantum theory with LSTM networks to learn emotional interactions between discourses. Ding et al. (2023) and Song et al. (2024) employed independent unimodal human annotations to simultaneously learn multimodal and unimodal representations using a multitask learning approach. However, this method requires independent unimodal annotations, and the high difficulty of manual annotation limits its widespread adoption. In response, Yu et al. (2021) proposed generating unimodal annotations automatically through algorithms. While this approach reduces the cost of creating unimodal labels, it is challenged by the instability and unreliability of the automatically generated labels, presenting a new bottleneck in research relying on unimodal emotion labels.

From the above research, it is evident that multimodal data analysis for emotion research across different modalities can effectively address the limitations of sparse data and incomplete information inherent in single-modal emotion analysis. By integrating data at various levels (Lian, Liu & Tao, 2021), the model’s performance is significantly enhanced, enabling more accurate emotion recognition. Additionally, targeted feature improvements within the foundational neural network can further augment model performance. Therefore, in this article, we employ the attention mechanism on top of the original feature fusion to achieve enhanced integration of features enriched with more modal information, facilitating emotion analysis through the fusion of multiple physiological signals.

Transformer model and multi head attention

The Multi-Head Attention Mechanism is a pivotal component of the Transformer model, enabling it to execute multiple parallel self-attention computations across different subspaces of the input, thereby allowing the model to attend to various positions or features. This mechanism enhances the model’s representational capacity by performing independent attention operations on multiple “heads” and subsequently concatenating the results. For an input vector X (typically an embedded representation of each word in a sequence), it is projected into three distinct vector spaces, producing matrices of queries (Q), keys (K), and values (V), as depicted in Fig. 1. These scores are divided by a scaling factor $\sqrt{d_k}$ to balance the range of values, and finally normalized to a probability distribution as follows:

(1) $$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(Q,K,V\right)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$where ${\mathrm{d}}_{\mathrm{k}}$ is the key vector dimension. Multiple independent self-attention computations are carried out in parallel by different heads, each utilizing its own set of linear transformation matrices for queries, keys, and values. Specifically, for a given head, the self-attention operation for the I head is computed as follows:

(2) $$\mathrm{h}\mathrm{e}\mathrm{a}{\mathrm{d}}_i=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(Q_i,K_i,V_i\right).$$

The outputs of all the heads are spliced and linearly transformed to get the final multi-head attention output:

(3) $$\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\mathrm{H}\mathrm{e}\mathrm{a}\mathrm{d}\left(Q,K,V\right)=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left(\mathrm{h}\mathrm{e}\mathrm{a}{\mathrm{d}}_1,\mathrm{h}\mathrm{e}\mathrm{a}{\mathrm{d}}_2,\dots ,\mathrm{h}\mathrm{e}\mathrm{a}{\mathrm{d}}_h\right)W^O$$where $W^O$ is the output linear transformation matrix and $\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(\cdot )$ denotes the result of splicing all the heads in dimension. As the attention mechanism continues to evolve, the Transformer model has been introduced and extensively adopted in time series analysis and natural language processing research. The model consists of two primary components: Encoder and Decoder, both structured with multiple layers of self-attention and feed-forward network layers.

The encoder consists of key components such as input embeddings, Multi-Head Self-Attention, a Feed-Forward Network (FFN), residual connections, and layer normalization. The decoder mirrors this structure but adds masking to restrict access to future information during word prediction. Its main components include Masked Multi-Head Attention, Encoder-Decoder Attention, and a Feed-Forward Neural Network, which is made up of two fully connected layers.

(4) $$\mathrm{F}\mathrm{F}\mathrm{N}\left(x\right)=\mathrm{m}\mathrm{a}\mathrm{x}\left(0,x{W}_1+b_1\right)W_2+b_2$$where $W$ represents weight matrix, $b$ is the bias vector, and $\mathrm{m}\mathrm{a}\mathrm{x}\left(\right)$ is the ReLU activation function. Considering that Transformer does not have a loop mechanism, the order information is introduced by position encoding. The calculation formula for position encoding is.

(5) $$\mathrm{P}{\mathrm{E}}_{\left(pos,2i\right)}=\mathrm{s}\mathrm{i}\mathrm{n}\left({\displaystyle \frac{pos}{{10000}^{2i/d_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}}}}\right)$$

(6) $$\mathrm{P}{\mathrm{E}}_{\left(pos,2i+1\right)}=\mathrm{c}\mathrm{o}\mathrm{s}\left({\displaystyle \frac{pos}{{10000}^{2i/d_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}}}}\right)$$where $pos$ is the position, and $d_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}$ is the embedding dimension. Following the decoder, the model uses a linear layer to map the hidden layer representation to a probability distribution over the vocabulary, ultimately generating the target word. Unlike traditional RNN and LSTM models, the Transformer’s parallel processing capability significantly boosts training efficiency. Furthermore, the multi-head attention mechanism enables the model to analyze time series data from multiple perspectives, enhancing its feature extraction capabilities. This allows the Transformer to perform exceptionally well in tasks like time series prediction and classification.

Feature extraction from multimodal data

The establishment for TAF-ATRM model

Following the detailed introduction of each component within the feature extraction module, this article presents the TAF-ATRM network, a multimodal physiological signal emotion recognition framework that integrates text, audio, and facial information. The network enhances the extracted features from text, audio, and facial expressions using the Transformer network, and subsequently fuses these features through the attention mechanism to achieve multi-level feature integration. This approach enables high-precision emotion analysis. The overall structure of the proposed framework is illustrated in Fig. 2.

In the TAF-ATRM framework, we adopt a combination of feature stacking and cross modal attention to fully integrate the multimodal information of text, speech, and image data. Firstly, we extract deep features of text, speech, and vision using BERT, MFCC+CNN, and ResNet respectively, and align the features through dimension transformation and temporal alignment to ensure that different modalities are in the same representation space. Subsequently, cross modal attention mechanisms are utilized for information exchange, enabling different modalities to focus on each other and reinforce their key features. Among them, text, speech, and visual features are respectively used as queries, keys, and values, and the correlation between modalities is calculated through multi head attention mechanism to capture their complementary information. After cross modal interaction, the features are concatenated to form a unified high-dimensional representation, which preserves the unique information of each modality and enhances the collaborative effect between modalities. Finally, the fused features are input into a fully connected classifier, which uses Softmax for sentiment classification prediction. Through this fusion strategy, we fully utilize the complementarity of multimodal data and combine it with the self attention mechanism of Transformer to effectively improve the accuracy and performance of sentiment analysis.

Dataset and experiment setup

In process of dataset selection, this article proposes the use of MOSI and MOSEI, two widely adopted datasets in multimodal sentiment analysis (Zhu, 2023; Wu et al., 2023; Yin et al., 2025). MOSI serves as a benchmark dataset for multimodal sentiment analysis, focusing on sentiment expression in video content. It comprises 93 opinion clips from YouTube, recorded by various speakers, each conveying strong emotional messages. MOSI provides data in three key modalities: textual (the speaker’s verbal content), phonetic (speech features), and visual (facial expressions). MOSEI (Multimodal Sentiment Analysis Dataset) is an expanded version of MOSI, designed for broader and more diverse multimodal sentiment analysis tasks. MOSEI includes video clips from over 1,000 speakers on YouTube, spanning a wide range of topics and sentiment expressions. Like MOSI, it provides data in text, speech, and visual modalities, with each sample labeled for both sentiment and emotion categories (e.g., happiness, anger, sadness). These datasets are valuable for cross-cultural sentiment analysis, as they contain videos from speakers across the globe, encompassing multiple languages, accents, cultural backgrounds, and diverse modes of emotional expression. They provide researchers with the opportunity to explore how emotions are conveyed across cultures, which is crucial for understanding cross-cultural emotion dynamics, especially in the context of multimodal interactions involving language, facial expressions, and speech features. Due to the fact that the MOSI and MOSEI datasets contain speakers from different regions and cultural backgrounds around the world, these videos cover multiple languages, accents, speech patterns, and facial expression differences, making them highly adaptable across cultures. These datasets can effectively reflect the expression of emotions in different cultural backgrounds, enabling the model to learn the commonalities and individualities of cross-cultural emotional features during training, thereby improving its generalization ability in multicultural environments and providing reliable data support for cross-cultural emotion recognition. By utilizing the MOSI and MOSEI datasets, researchers can train models to identify and analyze emotion features in varied cultural contexts, thereby enhancing the accuracy of cross-lingual emotion recognition and improving the comprehension of emotion transfer in cross-cultural communication.

Upon completing the dataset validation, further evaluation of the model’s performance is necessary. In this article, we analyze the results accuracy, precision, recall, and F1-score. We also compare our model against several widely recognized methods, including memory fusion network (MFN) (Zadeh et al., 2018), Recurrent Attended Variation Embedding Network (RAVEN) (Wang et al., 2019), Multimodal Cyclic Translation Network model (MCTN) (Hai et al., 2019), and Multimodal Transformer (MulT) (Delbrouck et al., 2020). Additionally, to investigate the effect of different modal information and feature fusion strategies, we conduct ablation experiments, analyzing variants such as T-ATRM, A-ATRM, and F-ATRM, which utilize only text, audio, or facial modalities, respectively, as well as TAF-TRM, which excludes feature enhancement.

Given that the model processes multimodal data, specific requirements for the experimental environment are necessary. The experimental environment configured for this study is outlined in Table 1.

In light of potential future applications and the role of sentiment analysis in cultural communication, this article focuses on three emotion classifications within the dataset: POSITIVE, NEGATIVE, and NEUTRAL. The aim is to realize sentiment classification within these three categories. The The key hyperparameters for the model is set in Table 2.

Method comparison and result analysis

After introducing the model, the relevant datasets, and the tasks addressed in this article, we performed an in-depth analysis of the model using two public datasets. We present the variation in loss during the training process, as well as the final recognition accuracy achieved on both datasets. Figure 3 illustrates the training loss process on the MOSI dataset.

In Fig. 3, it is evident that during the initial stages of training, the loss variation trends across the five models are relatively similar, suggesting comparable learning speeds on the dataset at the outset. However, as the number of iterations increases, the TAF-ATRM framework proposed demonstrates a clear advantage. The loss reduction rate of this framework is significantly faster than that of the other models, indicating that it captures the features in the data more efficiently, thereby optimizing the learning process. Eventually, the loss curve for TAF-ATRM stabilizes, suggesting that the model reaches a more stable state in the later stages of training.

In terms of accuracy for the three-class emotion recognition tasks, the TAF-ATRM framework also exhibits superior performance. The final recognition accuracy stabilizes at 0.863, notably outperforming the other models, demonstrating the higher accuracy and robustness in multimodal sentiment analysis. This framework effectively handles complex sentiment classification tasks. The results for the MOSEI dataset are presented in Fig. 4.

In Fig. 4, it is evident that the model’s loss variation trend during training closely correlates with its performance on the MOSI dataset. Given that this dataset contains a larger volume of data and more diverse sentiment expressions, the model’s complexity increases during training, leading to a relatively slower decline in loss. This reflects the challenges faced when handling large-scale, heterogeneous data. Nonetheless, the model proposed continues to exhibit stable performance, ultimately achieving an accuracy of 0.849 on the MOSI dataset. Although this is slightly lower compared to smaller datasets, it still underscores the model’s strong sentiment recognition capabilities on larger datasets. This result further validates the model’s effectiveness in managing complex multimodal sentiment analysis tasks, maintaining a high recognition rate despite the increase in data volume.

To more clearly compare the sentiment recognition performance of different models, we present additional comparison metrics under both datasets. The results of other model performance evaluation metrics are shown in Figs. 5 and 6, as well as in Tables 2–4.

By comparing the three key metrics across the two datasets, it is evident that the TAF-ATRM model performs exceptionally well in the emotion recognition task, demonstrating a strong balance between accuracy and robustness. The experimental results show that the model maintains consistent performance across datasets of varying sizes. On the smaller dataset, the TAF-ATRM model achieves F1-score of 0.866, while on the more complex and large-scale dataset, the F1-score remains high at 0.849. This trend mirrors the accuracy metric, highlighting the model’s stable classification ability and robustness across different dataset scales.

The results further validate the effectiveness of the proposed method, demonstrating its exceptional performance in emotion recognition on smaller datasets while also sustaining high accuracy and consistent results on larger, more complex datasets. This underscores the model’s strong generalization capability and its ability to achieve balanced recognition, even when tackling more challenging multimodal sentiment analysis tasks.

At the same time, we provided the corresponding ROC curves for two models, as shown in Fig. 7.

According to the results shown in Fig. 7, it can be seen that the overall change of the proposed method in the ROC curve is relatively average. The AUC values of TAF-ATRM in both datasets also change relatively evenly, with overall AUC values of 0.688 and 0.726, respectively. Compared with other models, not only is precision better, but the AUC training process is smoother, which can effectively achieve emotion classification and recognition research.

Ablation experiment and practical test analysis

To more thoroughly analyze the impact of different modal information and the integration of various attention modules on model performance, we conducted ablation experiments. The naming conventions for the models used in these experiments were introduced in “Dataset and experiment setup”. Additionally, to better address the issue of multimodal sentiment analysis in cross-cultural communication, we further extracted data from diverse cultural backgrounds. Specifically, we divided the cultural regions based on IP addresses, and then tested the accuracy of sentiment analysis for individuals from each region using the TAF-ATRM model proposed. This allowed us to compare the framework’s effectiveness across different cultural contexts.

Given the size of the dataset and the necessity for cross-cultural comparison, this section focuses exclusively on the MOSEI dataset. The comparison results of the ablation experiment are provided below, offering insights into how the inclusion or exclusion of specific modal information and attention modules affects model performance across cultures.

In Fig. 8, it is evident that in the single-modal emotion recognition process, even with the addition of the Transformer’s feature enhancement module, the overall emotion recognition performance remains suboptimal. The highest emotion recognition accuracy using only facial expression features is 0.793, falling short of 0.8, while the accuracy achieved with only voice features is 0.697, not exceeding 0.7. In contrast, after fusing the three types of modal information and removing the feature attention module, the model’s overall performance surpasses 0.8 in recognition accuracy. This demonstrates the critical role of multimodal fusion in enhancing emotion recognition.

The results for precision, recall, and F1-score in the right subgraph mirror the accuracy outcomes, further validating that different modalities provide essential complementary information during the emotion recognition process. Moreover, the use of attention-enhanced features significantly improves the model’s recognition accuracy, confirming that attention mechanisms contribute to refining feature representations and boosting performance.

After completing the ablation experiments, we conducted a deeper analysis of the TAF-ATRM framework’s ability to recognize emotions across different cultural contexts. To explore the model’s performance in cross-cultural sentiment analysis, we divided the data by continents and compared the model’s detailed performance against classical models and ablation models. As shown in Fig. 9, the accuracy statistics reveal minimal differences in overall model performance across different cultures. However, in terms of sentiment recognition performance, the TAF-ATRM model demonstrates a clear advantage. Whether considering the highest recognition accuracy or the mean accuracy, the TAF-ATRM model consistently outperforms the others.

This indicates that the framework exhibits stronger generalization and robustness when tackling emotion recognition tasks in various cultural contexts. Specifically, TAF-ATRM excels at capturing subtle emotional nuances across cultures while maintaining stable, high performance in diverse data environments. These results suggest that the TAF-ATRM model is not only well-suited to small-scale datasets but also delivers excellent results in large-scale, culturally diverse contexts. This finding offers strong support for future research in cross-cultural sentiment analysis, especially for complex tasks involving global, multicultural scenarios. Furthermore, the TAF-ATRM framework holds significant potential for real-world applications and provides new avenues for enhancing and optimizing cross-cultural sentiment recognition models.