Hybrid-Module Transformer: enhancing speech emotion recognition with HuBERT, LSTM, and ResNet-50

Introduction

Speech emotion recognition (SER) is like a puzzle where we try to figure out how someone feels by listening to their voice. It is not easy because people show their feelings in many ways when they talk. Thus, researchers have proposed many methods to deal with this challenge.

Traditional methods involve hidden Markov models (HMMs), which are suited for time-series data and widely used in the fields, including speech recognition, natural language processing, and bioinformatics (Mao et al., 2019; Feng & Narayanan, 2024; Dwivedi et al., 2022; Akbulut, Perros & Shahzad, 2020). Mao et al. (2019) focus on the development and evaluation of HMM-based architectures for utterance-level speech emotion recognition. They propose three HMM-based architectures for speech emotion recognition: Gaussian Mixture Model–Hidden Markov Model (GMM-HMM), Subspace Gaussian Mixture Model–Hidden Markov Model (SGMM-HMM), and Deep Neural Network–Hidden Markov Model (DNN-HMM). In the experimental evaluation, various HMM-based architectures were assessed on the CASIA corpus, achieving a maximum accuracy of 86.88%. However, HMMs rely heavily on hand-crafted features, which require expert knowledge and may not capture all the details of the data. Besides this, they cannot capture intricate patterns and relationships in the data.

Researchers have proposed deep neural networks (DNNs) as a means to learn complex patterns in data. These networks are effective for both feature extraction and classification (Trinh Van et al., 2022; Wani et al., 2021; Fahad et al., 2021). Compared to HMMs, DNNs are better at recognizing intricate patterns and relationships within the data, making them particularly effective for SER tasks. For example, in the work of Fahad et al. (2021) they combine the excitation source features and the Mel-frequency cepstral coefficients (MFCC) features to develop an emotion recognition system called DNN-HMM. The experimental results show that the epoch feature set is complementary to the MFCC feature set for emotion classification. The average emotion recognition rate of the proposed model using epoch features is 54.52% (Fahad et al., 2021). DNNs offer a robust and adaptable framework for modeling complex data, but they fail to process information from previous inputs, which is essential for time series prediction and language modeling.

Recurrent neural networks (RNNs) are more effective than traditional DNNs when dealing with sequential data because they can retain information over time. This capability enables RNNs to process inputs of varying lengths and to take into account the context and dependencies within a sequence. As a result, RNNs are well-suited for tasks such as language modeling, speech recognition, and time series prediction (Yadav et al., 2022; Trinh Van et al., 2022; Kons et al., 2022; Jermsittiparsert et al., 2020). For example, Trinh Van et al. (2022) present the results of speech emotion recognition with the Interactive Emotional Dyadic Motion Capture Database (IEMOCAP) corpus. In their case, three deep neural network models, convolutional neural network (CNN), convolutional recurrent neural network (CRNN), and gated recurrent unit (GRU), were used for emotion recognition. The results show that the proposed model gave the highest average recognition accuracy of 97.47% (Trinh Van et al., 2022). However, RNNs have difficulty learning long-range dependencies due to two main issues: the vanishing gradient problem, where gradients become too small to effectively update weights, and the exploding gradient problem, where gradients grow excessively large and destabilize the network.

Research shows that using self-supervised training on audio data enables full utilization of large volumes of unlabeled audio for learning latent features. This reduces the need for data labeling and associated costs (Zhao & Zhang, 2022; Chang et al., 2021; Chiu et al., 2022). Wav2Vec2, a transformer-based model trained with self-supervised methods, has offered several advantages over RNN-based models in SER tasks (Yi et al., 2020; Jain et al., 2023; Shahgir, Sayeed & Zaman, 2022). For example, Yi et al. (2020) applied the pre-trained wav2vec2.0 model to address the low-resource SER task. The model achieved over 20% relative improvements in six languages compared to previous work (Yi et al., 2020). Wav2Vec2 offers a combination of speed, efficiency, and accuracy that makes it a preferred choice for modern SER systems. While Wav2Vec2 is a powerful model for SER, it does have limitations, such as the need for massive computational resources and reliance on the availability of unlabeled data. These limitations highlight the need for further research and improvements to the SER model.

In recent years, Transformers have become a trend in SER tasks (Hazmoune & Bougamouza, 2024; Andayani et al., 2022; Wagner et al., 2023; Triantafyllopoulos et al., 2022). The Transformer architecture, introduced by Vaswani et al. (2017), represents a significant advancement in the field of sequence modeling, offering several advantages over traditional RNNs, DNNs, and HMMs. Unlike RNNs, which process data sequentially, Transformers are effective in capturing long-range dependencies. They are not constrained by the limitations of sequential processing, which allows for more efficient parallel computation and a significant reduction in training times. This advantage is beneficial for utilizing modern hardware capabilities, such as GPUs. Roy et al. (2021) propose the Routing Transformer, which utilizes content-based sparse attention inspired by non-negative matrix factorization. Unlike local attention models, the Routing Transformer does not rely on fixed attention patterns while maintaining a similar space-time complexity. Additionally, compared to previous approaches to content-based sparse attention, it does not require the computation of a full attention matrix. Instead, it selects sparsity patterns based on content similarity (Roy et al., 2021).

Recent advances have explored dynamic-scope Transformer architectures that allow models to adapt their attention span based on input characteristics (Chen et al., 2023a; Wang et al., 2023, 2024; Chen et al., 2023b). Examples include the Deformable Speech Transformer (DST) (Chen et al., 2023b), which learns token-specific window sizes; and the Time-Frequency Transformer (Wang et al., 2023), which models temporal and spectral attention separately before fusing them. These models share a core philosophy: letting the data guide the scale and focus of attention, rather than relying on fixed windows. While they improve performance on benchmarks such as IEMOCAP and Multimodal EmotionLines Dataset (MELD), they also introduce higher model complexity. For instance, DST achieves up to 71.8% weighted accuracy (WA) and 73.6% unweighted accuracy (UA) on IEMOCAP, but its dynamic modules require careful regularization to avoid overfitting on smaller datasets, highlighting a trade-off between flexibility and efficiency.

Capsule networks (CapsNets) have also emerged as a promising direction in SER due to their ability to preserve hierarchical relationships between features. Unlike traditional CNNs that lose spatial relationships during pooling, capsule networks use dynamic routing to retain part-whole relationships, making them particularly effective for capturing complex emotional cues in speech. Recent studies have explored their potential in SER tasks (Zhang et al., 2024, 2025). For example, Zhang et al. (2024) proposed a capsule-enhanced neural network (CENN) that integrates multi-head attention, residual blocks, and capsule layers. Their model achieved 72.88% accuracy on the Interactive Emotional Dyadic Motion Capture Database (IEMOCAP) dataset, showcasing the advantages of capsule-based architectures in capturing fine-grained emotion-related features (Zhang et al., 2024). However, CENN exhibits high computational complexity and performs less effectively on smaller datasets like Surrey Audio-Visual Expressed Emotion Database (SAVEE), highlighting the trade-off between representational power and model efficiency.

Inspired by these works, this article presents an idea to enhance SER models through the integration of various techniques, including HuBERT, LSTM, and ResNet (Yi et al., 2020; Jain et al., 2023; Hattori & Tamura, 2023; Li, 2021). Traditional approaches often struggle with either extracting meaningful features from raw audio, modeling long-range temporal dependencies, or recognizing hierarchical spatial patterns. The proposed model uses HuBERT’s self-supervised learning to extract richer and more discriminative audio representations without requiring large labeled datasets, leverages the sequential processing capabilities of LSTM to capture long-range dependencies, and incorporates convolutional layers inspired by ResNet-50 for effective pattern recognition. This hybrid design is motivated by the need to unify strengths across domains—self-supervised learning, temporal modeling, and spatial feature extraction—to form a more robust SER framework. This combination leads to a system that is more accurate and flexible for SER tasks. Additionally, it is capable of generalizing with limited labeled data, enhancing the overall performance and robustness of speech recognition tasks. The proposed method has been evaluated on two widely-used benchmark datasets for speech emotion recognition: Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) and MELD. On RAVDESS, which features high-quality emotional speech samples in a controlled environment, the model achieves a maximum accuracy of 76% and a precision of 78%. On the more challenging MELD dataset, which contains multi-party conversations with rich emotional diversity, the model attains an accuracy of 72.9% and a precision of 72.3%. Compared to existing approaches, including AlexNet, ResNet-50, and ResNet-101, our Hybrid-Module-Transformer model demonstrates superior performance across both datasets.

The main contributions of our article are as follows:

• We have developed an integrated Transformer model, which is designed for SER tasks. The hybrid model leverages the strengths of HuBERT for robust feature extraction and LSTM for capturing the temporal dynamics of emotions, leading to enhanced accuracy and contextual understanding. This integrated approach allows for better generalization across different speakers and emotional states. Compared to Wav2Vec2, HuBERT focuses on learning high-level representations of unmasked inputs to accurately infer the targets of masked ones, leading to an improvement in low-resource scenarios where Wav2Vec2 may struggle.

• The HuBERT module serves as the feature extractor, leveraging its Transformer architecture to extract robust features from raw speech data. HuBERT is trained on a large amount of unlabeled audio data using a self-supervised learning approach, allowing it to learn complex patterns and representations of speech.

• The LSTM module effectively extracts features while managing temporal dynamics, offering considerable advantages for processing speech data. This approach not only enhances the precision of emotion recognition across a spectrum of emotional states but also improves classification accuracy. By utilizing LSTM’s capability to capture speech patterns and understand the sequential flow of emotions over time, this model delivers reliable and detailed results for SER tasks.

• ResNet-50 is traditionally used for image recognition, but we applied its convolutional layers for feature extraction in audio processing. This approach allows the model to learn complex hierarchical features from audio data, capturing both low-level and high-level characteristics. This is especially beneficial in speech classification tasks, where the frequency content of speech signals is important for identifying different languages or emotional states.

• The hybrid Transformer model demonstrates exceptional performance in SER tasks, surpassing traditional models such as LSTM and ResNets. Experiments conducted on the RAVDESS Emotional Speech Audio dataset achieved an accuracy rate of 78%.

To demonstrate the advancements brought by our proposed approach, we compare its performance with several representative SER models reported in the literature. The baseline models include traditional architectures such as SGMM-HMM and DNN-HMM, as well as deep learning-based models like gated recurrent unit (GRU). In addition, we include two recent and competitive models: the Deformable Speech Transformer (DST), which represents the adaptive-attention Transformer family, and the CENN, which incorporates capsule structures for improved spatial relationships. While these models have shown impressive results on the IEMOCAP and Berlin Emotional Speech Database (EMODB) datasets, our Hybrid-Module-Transformer model achieves competitive performance on both RAVDESS and MELD, demonstrating strong generalization across diverse SER scenarios. Table 1 presents a quantitative comparison of these models, highlighting their features, advantages, and disadvantages. This comparison further motivates the hybrid design of our model, which balances the strengths of HuBERT, LSTM, and ResNet-50 to enhance speech emotion recognition.

This article is organized as follows: We first introduce our proposed model in ‘Methodology’. In ‘Implementation and Experiments’, we evaluate the proposed model using several evaluation metrics. After assessing the model’s performance, we summarize our findings and discuss future work in ‘Materials and Methods’.

Methodology

In this study, we propose a hybrid approach for feature extraction in SER by leveraging the complementary strengths of HuBERT-LSTM and ResNet-50. The process of speech emotion recognition in our model is shown in Fig. 1.

First, the original raw audio data is transformed into a 3-s Log-Mel Spectrograms. This transformation converts the audio signal into a format that is more manageable for our model. After the feature extraction, the Log-Mel Spectrograms is fed separately into the HuBERT-LSTM and ResNet-50 modules. The output from the ResNet-50 module consists of learned features that capture various aspects of the audio data. Next, the HuBERT model is employed to capture both the content and context of the audio. These features are then input into an LSTM network, which models the sequential dependencies in the audio data. Finally, all the feature sets are concatenated and input into a Transformer module for SER classification. By integrating these components, our model leverages the strengths of each module, enhancing overall system performance and leading to more accurate and robust emotion classification results.

ResNet-50 module

In order to better extract rich, hierarchical features from the Log-Mel Spectrograms, we have adopted ResNet-50 for audio processing. As shown in Fig. 2, the adapted ResNet-50 module consists of the following steps:

• Audio to spectrograms: Since ResNet-50 was originally designed for image recognition tasks, we convert raw audio signals into Log-Mel Spectrograms, which are 2D representations of the audio’s frequency content over time. This transformation makes audio data compatible with ResNet-50.

• Convolutional layers for feature extraction: The model utilizes the initial layers of ResNet-50 to extract low-level features from the spectrograms. These layers are effective at capturing local patterns and textures within the spectrograms. Each convolutional layer applies a set of filters to the input to extract features. The output of a convolution operation can be represented by the Eq. (1). Where F is the feature map, I is the input audio (Log-Mel Spectrograms), K is the kernel (filter), and * denotes the convolution operation. (1) $$F(x,y)=(I\ast K)(x,y)=\sum _m\sum _{n}I(m,n)\cdot K(x-m,y-n).$$

• Batch normalization: After convolutional layers, we apply batch normalization to normalize the activations. The normalized output is described by Eq. (2). In the equation, z represents the input to the batch normalization layer, $\mu$ is the mean of z, ${\sigma }^2$ is the variance of z, and $\epsilon$ is a small constant helps to prevent division by zero. After convolutions, Rectified Linear Unit (ReLU) activation functions are used to learn complex patterns in the audio data that indicate different emotions. (2) $$\hat{z}=\frac{z\text{-}\mu }{\sqrt{{\sigma }^2+\epsilon }}.$$

• Max pooling: A max pooling layer is employed to reduce the spatial dimensions of the feature maps, emphasizing the most significant elements. We use a 3 × 3 pooling window that moves across the feature map with a stride of 2. The window captures the highest value within its area, discarding the rest, and this process is repeated across the map. Max pooling reduces the network’s parameter count and computational load, thereby enhancing efficiency.

HuBERT-LSTM module

The HuBERT-LSTM architecture for SER tasks combines the strengths of self-supervised learning and sequence modeling to recognize emotions from speech signals effectively. Detailed workflow and architecture of the HuBERT-LSTM are shown in Fig. 3.

First, the raw audio data is transformed into Log-Mel Spectrograms and then fed into the HuBERT model. This model has been trained in a self-supervised manner, allowing it to learn robust speech representations without requiring labeled data. The HuBERT model outputs a sequence of embeddings that capture the acoustic features of the speech signal.

Next, these embeddings are processed by an LSTM network, which is effective at capturing long-term dependencies within sequence data due to its gating mechanisms. The workflow within an LSTM unit can be described by the following steps:

• – Forget gate: The forget gate determines how much of the previous cell state should be retained. As shown in Eq. (3), $\sigma$ represents the sigmoid function, $W_f$ are the weights associated with the forget gate, $h_{t-1}$ is the hidden state from the previous time step, $x_t$ is the input at time step $t$, and $b_f$ are the biases. (3) $$f_t=\sigma \left(W_f\cdot \left[h_{t-1},x_t\right]+b_f\right).$$

• – Input gate: The input gate controls how much of the new input will be added to the cell state, while the candidate cell state represents the new information to be stored. This is expressed in Eq. (4). Here, $i_t$ denotes the input gate activation, $\tilde{c}$ is the candidate value for the cell state, and $W_i$, $W_C$, $b_i$, and $b_C$ are the weights and biases for the input gate and candidate cell state, respectively. (4) $$\begin{array}{c}{i}_t=\sigma \left(W_i\cdot \left[h_{t-1},x_t\right]+b_i\right)\hfill \\ {\tilde{C}}_t=\tanh \left(W_C\cdot \left[h_{t-1},x_t\right]+b_C\right).\end{array}$$

After the LSTM processes the sequence, the output is passed through a fully connected layer and a softmax layer to classify the input audio into different emotional categories. The softmax function is given by Eq. (5). Where $z_k$ is the output of the fully connected layer for class $\kappa$, and K is the total number of emotion classes.

Finally, the outputs from HuBERT-LSTM and ResNet-50 are fed into a Transformer model for SER outputs. This combines the sequential information captured by LSTM with the spatial features extracted by ResNet-50. This combined architecture leverages HuBERT’s ability to extract meaningful acoustic features and LSTM’s capability to model the temporal dynamics of speech, providing a practical framework for recognizing emotions in speech signals.

(5) $$P(y=k|x)=\frac{\exp (z_k)}{{\sum }_{j=1}^K\exp (z_j)}.$$

Implementation and experiments

In this experiment, we conducted a comparative analysis and ablation studies with traditional SER models, including AlexNet, ResNet50, ResNet101, LSTM, and LSTM-HuBERT. Our study utilized the RAVDESS Emotional Speech Audio Dataset, which is well-known for its comprehensive emotional annotations and diverse vocal expressions, serving as the foundation for our experiments.

The comparative experiments aimed to benchmark the performance of our proposed model against several commonly used SER models. We evaluated the models using standard metrics, including accuracy, precision, recall, and the F1-score. These metrics allowed us to perform a detailed assessment of each model’s efficiency.

Additionally, we conducted an ablation study to assess the impact of various model components. By selectively disabling or modifying certain features, we were able to isolate the contributions of individual elements to the overall performance of the model. This process was instrumental in identifying which aspects of our model were most influential in achieving good performance in emotion recognition tasks.

Ablation study

First, we examine the impact of different modules in our model. As shown in Table 6, the “Full Model” serves as the baseline, demonstrating strong performance with a precision of 0.7727, Recall of 0.7852, accuracy of 0.793, and an F1-score of 0.789. These results indicate a well-balanced trade-off between precision and recall.

Table 6:

Ablation study results for our model.

Model	Precision	Recall	Accuracy	F1-score
Full model	0.7727	0.7852	0.793	0.789
w/o transformer attention	0.7689	0.7714	0.7929	0.7820
w/o ResNet-50	0.7538	0.7518	0.7819	0.7665
w/o transformer and ResNet-50	0.7538	0.756	0.7741	0.7649

DOI: 10.7717/peerj-cs.3292/table-6

The ablation study reveals the significance of each component for the model’s performance. When the Transformer Attention mechanism is removed (“w/o Transformer Attention”), there is a slight decrease across all metrics, with precision dropping to 0.7689 and recall to 0.7714. This suggests that attention plays a crucial role in capturing relevant features for emotion recognition. When ResNet-50 is excluded (“w/o ResNet-50”), we observe a drop in precision (0.7538) and accuracy (0.7819), highlighting its importance in stabilizing the learning process and enhancing generalization. Additionally, omitting ResNet-50 leads to a marginal decrease in recall (0.7518), indicating its contribution to preventing overfitting.

Removing both the Transformer and ResNet-50 from the model results in a global decline in performance. Precision drops to 0.7538, recall to 0.756, and accuracy to 0.7741. Furthermore, the F1-score decreases to 0.7649. These results show how various components contribute to enhancing the model’s overall performance.

In summary, the results show the role of each component in the model’s architecture and their contribution to the performance of the Full Model. The relatively high performance of the Full Model compared to its ablated versions supports the design of the proposed architecture in achieving effective results in SER.

The impact of removing different modules from the proposed architecture on classification precision is evident in Fig. 4. For instance, when the ‘sad’ emotion classification is analyzed, there is a drop in accuracy from 0.85 in the full model (d) to 0.8 in the model without the Transformer (c), and further down to 0.7 in the model without ResNet-50 (b).

When both the Transformer and ResNet-50 are omitted (as shown in (a)), there is a substantial decrease in accuracy across multiple emotion classifications. The ‘fear’ emotion classification, in particular, drops from 0.88 in the full model (d) to 0.86 in the model without the Transformer and ResNet-50 (a).

Removing the Transformer (c) also leads to decreased accuracy in classifying the ‘sad’ and ‘angry’ emotions, which fall from 0.85 and 0.78 in the full model (d) to 0.8 and 0.72, respectively. The ‘fear’ classification similarly decreases from 0.88 to 0.86.

The inclusion of both the Transformer and ResNet-50 (as in the full model (d)) results in an improvement in performance, with enhancements in the classification accuracy of ‘fear’ and ‘sad’ emotions. The ‘sad’ classification, in particular, benefits from the full model architecture, with a high accuracy of 0.85 compared to others.

In summary, the confusion matrices and their corresponding accuracy rates illustrate the significant contribution of each component to the model’s performance in correctly classifying emotional states. The data underscores the importance of an integrated model architecture that includes both the Transformer and ResNet-50, as seen in the full model (d), to achieve optimal performance in emotion recognition tasks.

Materials and Methods

Conclusion

This article presents a Hybrid Module Transformer model for speech emotion recognition, demonstrating its effectiveness even with limited data. By combining HuBERT, LSTM, and ResNet-50, our model excels in both feature extraction and emotional classification.

Experiments conducted on the RAVDESS dataset achieved an impressive accuracy rate of 78%, highlighting the robustness of the model in low-data situations. Furthermore, the model achieved an accuracy of 72.9% on the MELD dataset, demonstrating its robustness in more complex, real-world conversational scenarios involving multimodal emotional expressions. While our Hybrid Module Transformer model demonstrates promising SER results, a major limitation is its dependence on substantial computational resources, which could limit its applicability in low-resource environments. In future work, we plan to improve the model’s accuracy by exploring additional features and refining its architecture. We also aim to expand its capabilities to recognize a broader range of emotions and adapt to new languages with minimal retraining.

Code repository URL

The experimental code has been uploaded to GitHub, and the URL of the code is: https://github.com/qpqpaall/ser-main.

Third-party data

This study utilizes publicly available third-party datasets for training and evaluation. The datasets and their respective access links are listed below:

• RAVDESS: https://zenodo.org/records/1188976.

• MELD: https://github.com/declare-lab/MELD.

Supplemental Information