Emotion classification using advanced neural networks on sentence-level data

Dataset description

The dataset used in this study contains 422,746 labeled text samples categorized into six emotion classes: Joy, Sadness, Anger, Surprise, Fear, and Disgust. The distribution of the dataset is highly imbalanced, with Joy comprising 33.8% of the total samples and Disgust accounting for only 2.3%. This imbalance introduces challenges in model training and evaluation, as models often prioritize majority classes while underperforming on minority ones (Wahyuningsih & Chen, 2024). To mitigate this issue, preprocessing strategies and specialized modeling techniques were applied to ensure balanced performance across all emotion classes (Sangsawang, 2024a). The data used in this study can be accessed at this link: https://www.kaggle.com/datasets/kushagra3204/sentiment-and-emotion-analysis-dataset.

Preprocessing pipeline

To prepare the dataset for model training, a preprocessing pipeline was implemented. Initially, text cleaning was performed to remove special characters, numbers, and extra spaces, leaving only meaningful textual content (Hariguna, Sarmini & Azis, 2024). Sentences were then tokenized into individual tokens, represented as $T=\left\{t_1,t_2,\dots ,t_m\right\},$ where m is the number of tokens in the sentence. To ensure consistency, sequences were padded or truncated to a fixed length of 50 tokens. For sequences longer than 50, the first 50 tokens were retained ( $T^{\mathrm{\prime }}=T[1:L$]). For shorter sequences, padding tokens were added to achieve the required length ( $T^{\mathrm{\prime }}=T+{\left\{<PAD>\right\}}^{L-m}$) (Henderi et al., 2024). Finally, categorical emotion labels were encoded into numerical values, mapping each label y to an integer for compatibility with the neural network models (Fu et al., 2025).

Embedding methods

In this study, four types of embedding strategies were employed to represent textual data. The first was GloVe embeddings with 100 dimensions (glove.6B.100d.txt), which were pre-trained on large corpora and kept static (non-trainable) during model training. The second was FastText embeddings with 300 dimensions (cc.en.300.bin), which incorporate subword information to better handle out-of-vocabulary words (Kumar, 2024). The third approach utilized a trainable embedding layer with 100 dimensions, initialized randomly and updated throughout training to capture task-specific semantic features. Finally, BERT embeddings from the pre-trained bert-base-uncased model were applied, providing contextual representations that dynamically adapt to sentence-level context (Irfan, 2024b). It is worth noting that the dimensional differences between GloVe (100D) and FastText (300D) may influence performance outcomes; however, these configurations were retained to preserve the integrity of the original pre-trained models.

Model architectures

Three neural network architectures were evaluated in this study: BiLSTM, BiGRU, and BERT. Each architecture was optimized for emotion classification using specific configurations.

BiLSTM utilized bidirectional long short-term memory cells to capture both forward and backward dependencies in text. The hidden state at time t was computed as Eq. (1)

(1) $$h_t=LSTM\left(x_t,h_t-1,c_t-1\right),$$ $x_t$ is the input, $h_t$ is the hidden state, and $c_t$ is the cell state. The final representation was obtained by concatenating forward and backward hidden states use Eq. (2) (Sukmana & Khairani, 2024):

(2) $$h_{final}=concat\left(h_{forward},h_{backward}\right).$$BiGRU employed gated recurrent units, which simplified the computations by removing the cell state present in LSTMs. The hidden state was updated as Eq. (3)

(3) $$h_t=GRU\left(x_t,h_t-1\right),$$allowing the model to adapt dynamically to textual inputs (Sangsawang, 2024b).

BERT leveraged its transformer-based architecture, which relies on self-attention mechanisms to capture bidirectional contextual information. The self-attention mechanism was formulated as Eq. (4).

(4) $${\displaystyle \frac{alph{a}_{ij}=\exp \left(\left(WQ\ast e_i\right)\ast \left(WK\ast e_j\right)\right)}{su{m}_{k\left(\exp \left(\left(WQ\ast e_i\right)\ast \left(WK\ast e_k\right)\right)\right)}},}$$ $alph{a}_{ij}$ represents the attention weights between tokens $i$ and $j$, and $WQ,WK,$ and $WV$ are the query, key, and value weight matrices. The final representation of token $i$ was computed as Eq. (5) (Wahyuningsih & Chen, 2024):

(5) $$e_i^{\mathrm{\prime }}=su{m}_{j\left(alph{a}_{ij}\ast \left(WV\ast e_j\right)\right)}.$$

Training configuration and loss function

For model training, specific hyperparameter configurations were applied to ensure consistency and fair comparison across architectures (Yadav & Hananto, 2024). The BiLSTM models with GloVe and FastText embeddings, as well as the BiGRU model with trainable embeddings, were trained with a batch size of 64, learning rate of 0.001, and dropout rate of 0.5, each employing 128 hidden units over five epochs. In contrast, the BERT model was fine-tuned using a smaller batch size of 16, a learning rate of 5e−5, and a dropout rate of 0.1, also for five epochs (Sukmana & Oh, 2024).

All experiments were conducted on an MSI Cyborg 15 A12V laptop equipped with an Intel Core i7-12650H processor (10 cores), 16 GB DDR5 RAM, 512 GB NVMe SSD storage, and an NVIDIA GeForce RTX 4050 Laptop GPU with 6 GB VRAM, running on Windows 11. Training times varied across models, reflecting differences in computational complexity. On average, BiLSTM-GloVe required approximately 1.8 min per epoch (9 min for five epochs), BiLSTM-FastText 2.1 min per epoch (10.5 min total), and BiGRU 1.6 min per epoch (8 min total). By comparison, BERT was considerably more resource-intensive, averaging 6.5 min per epoch and totaling around 32.5 min for five epochs. These results highlight the trade-off between computational efficiency and predictive performance across different model architectures To train the models, the cross-entropy loss function was employed, which is defined as Eq. (6):

(6) $$L=-\left({\displaystyle \frac{1}{N}}\right)\ast sum\left(i=1toN,sum\left(j=1tok,y_{ij}\ast \log \left(y_{ha{t}_{ij}}\right)\right)\right).$$Here, $y_{ij}$ is the true label for the i-th sample and the j-th class, $y_{ha{t}_{ij}}$ is the predicted probability for the same, N is the number of samples, and k is the number of classes.

Evaluation method

The evaluation of model performance in this study was conducted through a rigorous and multi-dimensional approach, employing both quantitative metrics and qualitative error analysis to ensure a comprehensive assessment of emotion classification capabilities. Quantitatively, four primary evaluation metrics were utilized: accuracy, precision, recall, and F1-score. Model performance was assessed using several metrics (Doan, 2024). Accuracy was calculated as Eq. (7):

(7) $$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN},}$$

$TP,TN,FP$, and $FN$ are true positives, true negatives, false positives, and false negatives. Precision was defined as Eq. (8):

(8) $$Precision={\displaystyle \frac{TP}{TP+FP},\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{a}\mathrm{s}Recall={\displaystyle \frac{TP}{TP+FN}.}}$$

The F1-score, which balances precision and recall, was computed as Eq. (9).

(9) $$F1\text{-}score=2\ast {\displaystyle \frac{Precision\ast Recall}{Precision+Recall}.}$$

Accuracy reflects the proportion of correctly classified instances over the total number of instances, providing an overall measure of model performance (Wang et al., 2025). Precision quantifies the proportion of true positive predictions among all positive predictions, thereby assessing the model’s specificity in classifying each emotion class. Recall measures the proportion of true positive predictions relative to the actual number of positive instances, capturing the model’s sensitivity (Yadulla et al., 2024). F1-score, defined as the harmonic mean of precision and recall, was employed to balance the trade-off between these two metrics, particularly in the presence of class imbalance.

During model training, the cross-entropy loss function was adopted to quantify the discrepancy between predicted probabilities and true class labels. To address the inherent class imbalance within the dataset, a class-weighted loss function was implemented, ensuring that minority classes received proportionally higher penalties for misclassification, thereby promoting a more balanced learning process.

Furthermore, confusion matrix analysis was conducted to systematically examine the distribution of correct and incorrect predictions across all emotion classes. This analysis facilitated the identification of prevalent misclassification patterns, particularly between semantically overlapping categories such as “Joy” and “Surprise” or “Sadness” and “Fear”.

In addition to these evaluation metrics, hyperparameter optimization played a critical role in enhancing model performance. Strategies such as grid search and early stopping were employed to fine-tune parameters including learning rate, batch size, hidden layer dimensions, and dropout rates, thereby preventing overfitting and ensuring optimal generalization to unseen data (Pratama, 2024).

This comprehensive evaluation framework enabled a robust comparison of the BiLSTM, BiGRU, and BERT architectures, providing valuable insights into their respective strengths and limitations in the context of emotion classification.

Regularization technique

To mitigate the risk of overfitting and improve model generalization, dropout layers were incorporated across all neural architectures. For the BiLSTM and BiGRU models, a dropout rate of 0.5 was applied to the hidden layers, effectively preventing co-adaptation of neurons by randomly deactivating 50% of units during training. This technique is particularly useful in recurrent architectures, where long-term dependencies can otherwise lead to overfitting on training data (Cheng et al., 2025). In the case of BERT, a smaller dropout rate of 0.1 was used, consistent with standard fine-tuning practices for transformer-based models, as larger dropout values tend to degrade performance in pre-trained transformers.

Other commonly used regularization strategies were not implemented in this study but are acknowledged as promising directions for future work. Label smoothing, which prevents the model from becoming overconfident by redistributing a small portion of the probability mass across incorrect classes, could be beneficial for handling ambiguous emotion categories such as Joy–Surprise and Sadness–Fear. L2 weight decay, another widely adopted technique, applies penalties to large weight values during optimization, helping to reduce variance and improve robustness. Furthermore, text data augmentation methods, such as synonym replacement and back-translation, offer an effective way to expand training data and reduce sensitivity to lexical variations. These methods are particularly relevant in imbalanced datasets, where augmenting minority classes (e.g., Disgust) could help the model better generalize to underrepresented emotions.

Although not applied in the present study due to scope constraints, these regularization techniques remain important avenues for future exploration. Their integration has the potential to further enhance classification accuracy, improve robustness against overfitting, and increase the fairness of predictions across all emotion categories.

Reproducitbility

Both the dataset and the complete source code used in this study are publicly available at the following repository: https://doi.org/10.5281/zenodo.15009245. This ensures transparency and facilitates reproducibility of the experiments, allowing other researchers to replicate and extend the findings presented in this work.

Dataset overview

The dataset used in this study consists of 422,746 text samples distributed across six emotion labels: Joy, Sadness, Anger, Surprise, Fear, and Disgust. Among these, Joy represents the majority class with 33.8% of the samples, while Disgust, the minority class, accounts for only 2.3%. This significant imbalance presents challenges in training robust models, as they often prioritize performance on majority classes while neglecting minority ones.

Addressing this imbalance is crucial for achieving fair and generalized results. Techniques such as oversampling, class-weighted loss functions, or data augmentation could improve minority class performance. The distribution of the dataset is summarized in Table 1, providing a clear overview of the class imbalances.

Preprocessing and feature engineering

To prepare the dataset for modeling, a robust preprocessing pipeline was implemented. This began with text cleaning, removing special characters, numbers, and irrelevant symbols to ensure that the textual content was clean and meaningful. Next, tokenization split sentences into individual tokens, which were subsequently padded or truncated to a fixed length of 50 tokens to standardize input sizes for the neural network models.

Label encoding was applied to convert the categorical emotion labels into numerical values suitable for training. These preprocessing steps ensured uniformity in the data while preserving essential features. The preprocessing steps are summarized in Table 2.

This pipeline provided a solid foundation for applying neural network architectures and ensured compatibility with advanced models such as BERT.

Model configurations and performance

Three neural network architectures were evaluated: BiLSTM, BiGRU, and BERT. BiLSTM was tested with two pre-trained embedding methods: GloVe (100-dimensional) and FastText (300-dimensional). BiGRU was trained with trainable embeddings, allowing it to adapt specifically to the dataset. BERT, leveraging its transformer architecture, utilized pre-trained contextual embeddings for enhanced performance. Table 3 outlines the configurations of these models.

The performance metrics of these models are summarized in Table 4.

BERT achieved the best performance across all metrics, emphasizing the effectiveness of contextual embeddings. BiGRU with trainable embeddings outperformed both BiLSTM configurations, demonstrating the importance of fine-tuning embedding layers.

Hyperparameter tuning

Hyperparameter tuning was critical in optimizing the performance of each model. For BERT, adjustments to learning rate and batch size significantly influenced the model’s convergence and accuracy. The best performance was achieved with a learning rate of 5e−5, as shown in Table 6. Higher learning rates led to unstable training, while lower rates resulted in slower convergence.

For BiLSTM and BiGRU, the number of hidden units and dropout rates were varied to improve generalization. Increasing the number of hidden units to 128 resulted in slightly improved accuracy but increased computational cost. The addition of dropout (set at 0.5) proved essential in mitigating overfitting, particularly for models with trainable embeddings.

These findings underscore the importance of systematic hyperparameter optimization to unlock the full potential of each model.

Confusion matrix analysis

The confusion matrix for BERT, displayed in Table 7, reveals the model’s classification strengths and weaknesses. While BERT excelled in distinguishing well-separated classes such as Joy and Anger, it struggled with overlapping categories like Joy and Surprise or Sadness and Fear.

The matrix highlights areas where the model’s understanding of context may need improvement. Augmenting the dataset with more diverse examples and incorporating external knowledge bases could help reduce these errors.

Comparative analysis of embedding methods

The choice of embedding methods played a crucial role in determining the overall performance of the models. Pre-trained embeddings such as GloVe and FastText provided strong initial baselines, particularly for BiLSTM. GloVe, with its 100-dimensional representation, demonstrated effective generalization and computational efficiency, making it suitable for tasks with limited computational resources. On the other hand, FastText’s 300-dimensional embeddings utilized sub-word information, allowing the model to handle out-of-vocabulary words effectively. However, both embeddings are static, which limits their ability to adapt dynamically to the specific context of the dataset.

Trainable embeddings, as employed in BiGRU, offered significant flexibility by allowing the model to learn features directly from the dataset. This adaptability resulted in better performance compared to BiLSTM with static embeddings. Trainable embeddings provided the model with domain-specific nuances, which were particularly advantageous for improving accuracy and loss metrics.

BERT’s contextual embeddings outperformed all other methods by a significant margin. Unlike static or trainable embeddings, BERT dynamically generates token-level embeddings that consider the entire context of a sentence. This contextual understanding enabled BERT to capture intricate relationships within the text, making it exceptionally well-suited for handling nuanced and overlapping emotional expressions. As shown in Table 8, BERT achieved the highest accuracy (94.07%) and F1-score (94.05%), along with the lowest loss (0.0950), confirming its superiority.

The analysis highlights a clear trend: the more contextual and adaptable the embedding method, the better the model’s performance. Static embeddings, while efficient and robust for certain tasks, fail to capture the nuanced relationships in text data. Trainable embeddings bridge this gap by learning domain-specific representations but still lack the global contextual understanding provided by BERT.

Future work could explore combining these embeddings into hybrid models. For instance, using pre-trained embeddings as initialization for trainable layers or combining static embeddings with contextual embeddings could further improve performance. Additionally, contextual embeddings like those in BERT can be fine-tuned on domain-specific corpora to enhance their adaptability, especially for tasks with highly specialized language.