Fusing Transformer-XL with bi-directional recurrent networks for cyberbullying detection

Machine learning approaches

Hoque & Seddiqui (2023) extensively explored machine learning approaches for detecting cyberbullying. In the study, after comparing models like SVM, random forest (RF), long short-term memory (LSTM), and bidirectional encoder representations from transformers (BERT) to identify cyberbullying in Bengali; the BERT-based model has the greatest accuracy of 80.165%. The study, which is sponsored by Bangladesh’s ICT division, emphasizes the requirement of feature selection and pre-processing as well as the need of further study in low-resource languages. Social media elements increased prediction performance for all evaluated machine learning (Bozyiğit, Utku & Nasibov, 2021) have shown to be important in cyberbullying detection. Talpur & O’Sullivan (2020) produced features from Twitter content in a binary setting, using a pointwise mutual information strategy in a cyberbullying detection framework. The findings were encouraging concerning Kappa, classifier accuracy, and f-measure metrics. Cheng et al. (2020) proposed an unsupervised cyberbullying detection model using a time-informed Gaussian mixture model that beat state-of-the-art unsupervised methods and obtained competitive performance relative to supervised models.

Deep learning approaches

Saifullah et al. (2024) proposed BullyFilterNeT, which is a deep learning system designed for Bengali cyberbullying detection. According to the study, the BanglaBERT-based model achieves an astounding accuracy of 88.04%. In a similar vein, Jobair et al. (2023) investigate Bengali hate speech identification using LSTM, bidirectional long short-term memory (BiLSTM), gated recurrent unit (GRU), convolutional neural network (CNN), and BERT models. In the study, BERT shows 80% accuracy on a new dataset and 97% on an old dataset. These results highlight how well transformer-based deep learning models work to combat hate speech and cyberbullying in languages with limited resources. Moreover, Kumar & Sachdeva (2022b) integrates textual, visual, and info-graphic data to present a deep neural network for multimodal cyberbullying detection, which achieves remarkable performance with an area under the receiver operating characteristic (AUC–ROC) of 0.98. Finally, Das et al. (2021) uses an attention-based recurrent neural network to address hate speech detection in Bengali social media.

Hybrid approaches

Wahid & Al Imran (2023) proposed the multi-feature transformer and deep neural network for Bengali multi-dimensional cyberbullying detection. The researchers combine user profiles, lexical characteristics, contextual embeddings, and semantic linkages for better efficiency of the model. Iwendi et al. (2023), on the other hand, shows that deep learning architectures are effective in detecting cyberbullying. To improve cyberbullying detection in social media content, Kumar & Sachdeva (2022a) developed a Bi-GRU-Attention-CapsNet (Bi-GAC) model that uses sequential semantic representations and geographical location data. Paul, Saha & Hasanuzzaman (2022) also presents a deep learning-based multimodal framework and achieves encouraging results with a 0.75 F-measure on the Vine dataset by employing ResidualBiLSTM-RCNN architecture. In addition, Karim et al. (2021) provides DeepHateExplainer, an explainable model for Bengali hate speech identification that outperforms baselines from neural networks and conventional machine learning in identifying different kinds of hate speech.

Transformer approaches

Han et al. (2023) describe the LTAnomaly model, which combines a Transformer with LSTM for efficient anomaly identification and represents logs using semantic information and sequence links. Teng & Varathan (2023) evaluate conventional machine learning and transfer learning methods for cyberbullying detection by employing AMiCA data and methodical annotation procedures. Roy & Mali (2022) use transfer learning in text-based social media cyberbullying detection methods. Elsafoury et al. (2021b) examine attention weights and feature significance scores to clarify BERT’s effectiveness in detecting cyberbullying throughout a range of datasets. Elsafoury et al. (2021a) further suggest employing modern contextual language models such as BERT and slang-based word embeddings to provide better representations of cyberbullying-related datasets.

Finally, a survey of current research in cyberbullying detection and deep learning approaches applied to social media data emphasizes the field’s dynamic nature. This comprehensive analysis demonstrates that approaches for evaluating textual data from social media networks are always evolving and being refined. However, several obstacles remain, such as the need for larger and more diverse datasets, greater model interpretability, and increased algorithm scalability. These shortcomings provide an opportunity for future study into fresh techniques and solutions in cyberbullying detection.

The findings of this article provide the foundation for our proposed Fusion Transformer-XL architecture, which intends to contribute to bully categorization by utilizing advances in deep learning and cyberbullying detection approaches. We want to develop cyberbullying detection in social media situations by expanding on current knowledge and tackling highlighted limitations.

Informal definition

Cyberbullying detection is the process of identifying and categorizing cyberbullying incidents in digital communication systems. It entails identifying written information on the internet that displays abusive, harassing, or offensive behavior towards people or organizations.

Formal definition

The goal of cyberbullying detection is to create a system that can automatically recognize instances of cyberbullying in text data. The objective is to construct a predictive model $f(x_i)$ that can accurately assign labels to new, unseen text samples. The dataset D contains text samples $x_i$ and their corresponding labels $y_i$, where $x_i$ represents a piece of text and $y_i$ denotes its cyberbullying classification label (e.g., “Not bully,” “Troll,” “Religious,” “Sexual,” or “Threat”). The model’s goal is to distinguish between different types of cyberbullying behavior by identifying patterns and characteristics that indicate the use of abusive or harassing language. This will aid in the development of more secure online settings.

Proposed fusion Transformer-XL model training algorithm

Algorithm 1 displays the Fusion Transformer-XL model’s training process, which is intended for the identification of cyberbullying. The input parameters, which include the input shape, vocabulary size, embedding dimension, number of attention heads, and dropout rate, among others, are defined first by the algorithm. To build the architecture, the model function initializes an embedding layer initially, then iteratively adds numerous Transformer-XL blocks to extract contextual information from the input data. To estimate class probabilities for the target categories, the output from the Transformer blocks is then processed via a sequence of Dense, Bidirectional GRU, and LSTM layers, ending with a softmax output layer. The implementation of a structured strategy enhances the model’s capacity to identify intricate patterns within the data, hence improving its efficacy in cyberbullying detection assignments.

Datasets statistics

Table 1 presents a comprehensive summary of the Bengali and English datasets used in the research. The Bengali dataset has 44,001 texts with an average length of 75.51 words. The texts range from a minimum length of 0 words to a maximum length of 1,627 words. By contrast, the English dataset has 99,990 texts with an average length of 119.49 words. The largest length of a text is 763 words, while the shortest length is four words. Bengali has a larger standard deviation in text length (109.54) compared to English (77.14), suggesting a greater degree of variation in text lengths. In addition, the English dataset has a much greater number of distinct terms (270,439) and total words (2,132,882) in comparison to the Bengali dataset, which has 56,199 unique words and a total of 535,585 words. These statistics demonstrate the variations in the distribution of text length and the level of lexical diversity between the two datasets.

Data preprocessing

Stop words removing

The lack of a standardized list of stop words makes it particularly difficult to recognize and deal with stop words in the Bengali language (Mahmud, Ptaszynski & Masui, 2024). We used a manual method to purify datasets for insightful analysis, meticulously compiling a list of stop words using in-depth regular expression analysis.

Our curated list includes words such as (e.g., ’অজ’, ’অজগর’, ’অঞ’, ’অট’, ’অটল’, ’অড’, এমব’, ’এমর’, ’এর’, ’এরকম, ‘ৎব’, ’ৎস’, ’ৎসক’, ’ৎসকন) and more. These stop words are phrases that are often used but are judged to have negligible worth in a particular context. These were found after careful linguistic study. We had to rely on human analysis and create a custom stop word list that was suited to the peculiarities of Bengali because there was no pre-compiled list of stop words appropriate to the language. Consequently, a comprehensive refining procedure was applied to the dataset, removing all instances of these stop words. By doing this, we ensured that the dataset was free of irrelevant data, making it possible to examine the underlying data in a more meaningful and targeted manner. Our study is more accurate and pertinent because of this careful curation, which concentrates on important details within the framework of the Bengali language.

Class imbalance mitigation through random oversampling

To address the class imbalance in our dataset, we employed random oversampling, which aims to balance class representation (Saadi & Dhannoon, 2023). Class imbalance in machine learning occurs when certain classes are underrepresented, leading to biased model performance (Rafi-Ur-Rashid, Mahbub & Adnan, 2022). We utilized the RandomOverSampler function from the imbalanced-learn package, which randomly selects examples from minority classes and duplicates them until class distribution is balanced. Mathematically, the procedure can be described as follows:

$$N_{\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}=\sum _{i=1}^C{n}_i$$

$$N_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}=max(n_1,n_2,\dots ,n_C).$$

Here, $N_{\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}$ represents the total instances in the dataset, $n_i$ denotes the instances in class $i$, and C is the total number of classes. The oversampling process aims to increase the instances in minority classes to match the number in the majority class ( $N_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}$).

Data augmentation

Synonym replacement is a key technique in Bengali text augmentation that enhances vocabulary while preserving semantic meaning. Given an original Bengali text $T_{\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{g}}$ with $n$ words $w_i$, the goal is to substitute each word with a synonym if available:

$$T_{\mathrm{a}\mathrm{u}\mathrm{g}}={\left\{\begin{array}{cc}{s}_i& \mathrm{i}\mathrm{f}\mathrm{s}\mathrm{y}\mathrm{n}\mathrm{o}\mathrm{n}\mathrm{y}\mathrm{m}\mathrm{s}(w_i)\mathrm{e}\mathrm{x}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{s}\hfill \\ w_i& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}\right\}}_{i=1}^n.$$

Here, $s_i$ represents a randomly chosen synonym for the $i^{th}$ word $w_i$. Resources like Bengali WordNet (Majumder et al., 2022) are vital for identifying synonyms. If no synonyms exist, the original word is retained to maintain semantic coherence.

Given the linguistic diversity of Bengali, it is crucial to consider cultural and regional nuances during synonym substitution. For instance, the term “খুিশ” (khushi), meaning “happy,” can be replaced with synonyms like “আনন্িদত” (anondito) or “উৎসবগ্রস্ত” (utsobgrashto), each carrying distinct emotional connotations. Thus, synonym replacement not only broadens vocabulary but also captures the subtleties of the Bengali language.

Text tokenization

In this study, we developed tokenizers specifically for the Bengali language to handle a wide range of linguistic data efficiently. The BERT tokenizer was utilized to encode Bengali text entries into input IDs and attention masks. The tokenizer was pre-trained using Bengali language data (Sarker, 2020).

Following tokenization, each language dataset training, validation, and test dataset was given its distinct array, including the input IDs and attention masks. The arrays were transformed into NumPy arrays to make processing and manipulation efficient. Any extraneous dimensions were eliminated to optimize the input data format. The resultant tokenized input arrays were then prepared using BERT-based models. In this study, the dataset was divided into 70:15:15 ratios for training, testing, and validation, respectively.

Base line models

Hyperperameter settings

Table 2 summarizes our hyperparameter configurations for the models used in this research. Each row represents a distinct hyperparameter, such as embedding size, attention heads, feed-forward dimensions, and other related parameters, while the columns list the models, including Transformer, Transformer-GRU, Transformer-LSTM, Transformer-GRU-LSTM, Transformer-BiGRU-BiLSTM, Transformer-XL, and Fusion Transformer-XL. We select these hyperparameters to optimize performance while maintaining computational efficiency. The baseline models use an embedding dimension of 128, whereas we set Fusion Transformer-XL to 256 to enhance contextual representation in Bengali, which exhibits greater morphological complexity than English. We maintain two transformer blocks across all models to prevent overfitting while preserving adequate representational capacity. We adjust the multi-head attention mechanism to control training parallelization, setting Fusion Transformer-XL to two heads to ensure stable gradient propagation and reduce computational complexity. We set the feed-forward dimension to 256 in Fusion Transformer-XL to improve feature representation without excessive parameter expansion. Specific models incorporate BiGRU and BiLSTM layers to enhance contextual understanding, with 64 units selected for efficient sequence learning while maintaining computational feasibility. We utilize relative positional encoding in Transformer-XL and Fusion Transformer-XL to capture long-range dependencies, which are crucial for cyberbullying detection. We apply a dropout rate of 0.1 for most models to mitigate overfitting, while Fusion Transformer-XL uses 0.2 to regularize its larger parameter set. We set the memory length to 128 to ensure effective context retention. We use the Adam optimizer for all models except Transformer-BiGRU-BiLSTM, which employs Nadam for better adaptive learning rate adjustments. Fusion Transformer-XL adopts a batch size of 64 to accommodate its increased complexity while maintaining training stability. With 28.66 million trainable parameters, Fusion Transformer-XL significantly exceeds the baseline models, reflecting its enhanced capacity to capture cyberbullying patterns. These choices strike a balance between computational efficiency and performance, making our model suitable for Bengali and other low-resource languages. Future research should explore hyperparameter tuning and alternative architectures.

Training setup

In this section, we present a comprehensive assessment of our proposed Fusion Transformer-XL model for detecting cyberbullying in the Bengali language. We conduct the evaluation using computing resources such as an AMD Ryzen CPU, 16GB of RAM operating at a clock speed of 3,200 Hz, and an RTX GeForce 2060 GPU with 12GB of memory. We use Python version 3.11.9 and TensorFlow version 2.15.0 to refine the model’s parameters using methods such as Adam optimisation and early stopping regularisation. During the training process, we closely track important metrics such as accuracy, precision, recall, and F1 score to evaluate the effectiveness of the model. Additionally, we use k-fold cross-validation to ensure the model’s reliability and stability. By looking at the results of the experiment, we look into how hyperparameters like the number of transformer-XL blocks and attention processes affect the results. This will help us understand model’s efficiency and applicability in real-world settings for detecting cyberbullying in Bengali text data.

Evaluation metrics

The focus is on evaluating the efficacy of our suggested hybrid transformer-XL model via the use of several performance indicators. These measures, such as accuracy ( $Acc$), precision (P), recall (R), and F1-score ( $F1$), are crucial indications of the model’s prediction skills and its capacity to identify instances of cyberbullying in Bengali text data. Mathematically, accuracy is quantitatively determined by dividing the number of properly predicted cases by the total number of instances. $$Acc=\frac{TP+TN}{TP+TN+FP+FN}.$$

The variables TP, TN, FP, and FN represent true positives, true negatives, false positives, and false negatives, respectively. Precision is a measure that calculates the ratio of correct positive predictions to all positive predictions: $$P=\frac{TP}{TP+FP}.$$

While recall measures the proportion of true positive predictions among all actual positive instances: $$R=\frac{TP}{TP+FN}.$$

The F1-score, the harmonic mean of precision and recall, provides a balanced assessment of the model’s performance: $$F1=2\frac{PR}{P+R}.$$

The performance study highlights the hybrid transformer-XL model’s capacity to handle the complexity of Bengali cyberbullying detection. The model obtains a high F1-Score by skilfully balancing recall and accuracy, showing that it can reliably detect actual cases of cyberbullying in addition to making correct predictions. This thorough assessment shows that the model has the potential to be a dependable instrument for automated detection and intervention, which will help create safer online settings.

Comparison with baseline models

The performance of several neural network models on Bengali and English datasets is compared in Table 3, both with and without data augmentation. The available models are Transformer, Transformer-GRU, Transformer-LSTM, Transformer-GRU-LSTM, Transformer-BiGRU-BiLSTM, Transformer-XL, and Fusion Transformer-XL. Each model’s accuracy is tracked throughout training, validation, and testing. Data augmentation, as shown in the table provided, greatly enhances performance for all models. The bolded accuracy values in the table highlight the highest test accuracies achieved among the models on each dataset. Notably, the Fusion Transformer-XL model demonstrates superior performance, attaining a test accuracy of 98.13% and 99.65% on the Bengali and English datasets, respectively, when augmentation is used. In comparison, without augmentation, the model achieves accuracies of 88.36% and 99.00% for the same datasets. This demonstrates the efficacy of data augmentation in improving the overall performance of models and identifies the Fusion Transformer-XL as the most resilient model architecture for natural language processing tasks.

Table 4 presents a performance comparison of several models on Bengali and English datasets, with and without data augmentation. The models available include Transformer (T), Transformer-GRU (T-GRU), Transformer-LSTM (T-LSTM), Transformer-GRU-LSTM (T-GRU-LSTM), Transformer-BiGRU-BiLSTM (TGL), Transformer-XL (T-XL), and Fusion Transformer-XL (FTXL). Performance is evaluated using precision (P), recall (R), and F1-score (F1). Data augmentation greatly improves all metrics. The Fusion Transformer-XL model with augmentation (FTXL, Aug = 1) obtained the greatest scores: 98.13% precision (P), 98.11% recall (R), and 98.12% F1 score for Bengali; and 99.56% precision (P), 99.62% recall (R), and 99.59% F1 score for English. This underscores the efficacy of data augmentation in enhancing model precision and resilience.

Proposed fusion transformer-XL performance metrics

Figure 2 displays four confusion matrices that demonstrate the effectiveness of a fusion transformer model in recognising instances of cyberbullying in both Bengali and English languages. Figures 2A and 2B illustrate the model’s performance in Bengali, with (a) using data augmentation and (b) without it. In scenario (a), the model accurately categorises incidents as “Not Bully” 4,630 times, “Religious” 4,447 times, “Sexual” 4,468 times, “Threat” 4,410 times, and “Troll” 4,421 times. In scenario (b), where data augmentation is not used, the number of accurate classifications decreases to 1,397 for the category “Not Bully,” 1,465 for “Religious,” 1,186 for “Sexual,” 1,509 for “Threat,” and 1,214 for “Troll.” Figures 2C and 2D illustrate the model’s performance in English, comparing situations with and without data augmentation. In condition (c), the model attains accurate categorizations of 5,059 instances for “ethnicity/race,” 5,067 instances for “gender/sexual,” 15,079 instances for “not_cyberbullying,” and 4,686 instances for “religion.” In the absence of data augmentation in (d), the number of accurate categories decreases to 1,678 for “ethnicity/race,” 1,648 for “gender/sexual,” 5,077 for “not_cyberbullying,” and 1,496 for “religion.” The findings suggest that the use of data augmentation significantly improves the accuracy of the model’s classification in both languages, as shown by the higher number of right predictions (diagonal values) and lower number of misclassifications (off-diagonal values). The visual representations emphasize the vital importance of data augmentation in enhancing the efficacy of cyberbullying detection methods.

Figure 3 exhibits four receiver operating characteristic (ROC) curves that assess the efficacy of the fusion transformer model in identifying cyberbullying in both Bengali and English scenarios. Figure 3A demonstrates the efficacy of the model in detecting Bengali cyberbullying with data augmentation. It achieves flawless classification with an area under the curve (AUC) of 1.00 for all categories, including “Not Bully,” “Troll,” “Sexual,” “Religious,” and “Threat.” Figure 3B displays the model’s performance when data augmentation is not used. The AUC values for “Not Bully,” “Troll,” “Sexual,” “Religious,” and “Threat” are marginally decreased to 0.98, 0.97, 0.95, 0.99, and 1.00, respectively. The performance of the model in English cyberbullying detection with data augmentation is shown in Fig. 3C. It achieves an AUC of 1.00 across all classes, including “ethnicity/race,” “gender/sexual,” “not_cyberbullying,” and “religion.” Figure 3D displays the model’s performance when data augmentation is not used. It maintains an AUC of 1.00 for most classes but slightly decreases to 0.99 for the “religion” class. The ROC curves showcase the exceptional efficacy of the fusion transformer model, especially when combined with data augmentation. This demonstrates its strong capacity to reliably detect different types of cyberbullying in several languages.

Baseline models performance metrics

The confusion matrices in Fig. 4 illustrate how well different transformer-based models perform when it comes to identifying Bengali cyberbullying with data augmentation. A to F, the sub-figures, each represent a distinct model: Transformer in Fig. 4A, Transformer-GRU in Fig. 4B, Transformer-LSTM in Fig. 4C, Transformer-GRU-LSTM in Fig. 4D, Transformer-BiGRU-BiLSTM in Fig. 4E, and Transformer-XL in Fig. 4F. The distribution of genuine labels against predicted labels is depicted in the diagrams, giving a clear picture of each model’s accuracy and the many kinds of mistakes that might occur. This emphasizes how well data augmentation can enhance model performance.

With the use of data augmentation, Fig. 5 displays a number of confusion matrices that show how well different transformer-based models do in identifying English cyberbullying across a variety of categories, including religion, non-cyberbullying, gender/sexual orientation, and ethnicity/race. The results for the transformer model are shown in Fig. 5A, the transformer-GRU model in Fig. 5B, the transformer-LSTM model in Fig. 5C, the transformer-GRU-LSTM model in Fig. 5D, the transformer-BGRU-BiLSTM model in Fig. 5E, and the transformer-XL model in Fig. 5F. By comparing the real and predicted labels in each confusion matrix, the model’s precision in classifying cyberbullying cases with data augmentation is brought to light. The matrices indicate the number of correctly classified cases along the diagonal and the number of misclassified cases off-diagonal, shedding light on the relative efficacy of each model and the regions with higher rates of misclassification. Together, these visualisations highlight how different algorithms perform in terms of accurately recognising cyberbullying incidents in the enriched dataset.

A set of receiver operating characteristic (ROC) curves is shown in Fig. 6, which illustrates how well different transformer-based models identify Bengali cyberbullying in several classes when data is supplemented. The ROC curve for the transformer model is specifically shown in Figs. 6A–6F for the transformer, transformer-GRU, LSTM, transformer-GRU-LSTM, BGRU-BiLSTM and Transformer-XL models, respectively. Plotting the true positive rate vs. the false positive rate for each class—including threat, gender/sexual orientation, ethnicity/race, and not cyberbullying—is done using a ROC curve. The models’ performance is shown by the AUC values in the legend; greater AUC values correspond to better performance. When data augmentation is performed, the curves show that all models have high AUC values across the classes, demonstrating a significant capacity to discern between the various forms of cyberbullying. The combined impact of these visualisations is to show how different models differ in how well they can detect Bengali cyberbullying cases in the expanded dataset.

With data augmentation, Fig. 7 displays a series of ROC curves showing how well different transformer-based models identify English cyberbullying in various classes. The ROC curve for the transformer model is specifically shown in Figs. 7A–7F, for the transformer, transformer-GRU, LSTM, transformer-GRU-LSTM, BGRU-BiLSTM and Transformer-XL models, respectively. Plotting the true positive rate vs. the false positive rate for each class-ethnicity/race, non-cyberbullying, religion, and gender/sexual orientation-is what each ROC curve does. The models’ performance is shown by the AUC values in the legend; greater AUC values correspond to better performance.

According to the study, transformer-based models perform noticeably better at identifying cyberbullying in both Bengali and English datasets when data augmentation is used. Across categories including Not Bully, Religious, Gender/Sexual content, and Ethnicity/Race, precision, recall, and F1-score, as well as confusion matrices and ROC curves, demonstrate notable increases in model accuracy and dependability. Accurately identifying objectionable content is typically a greater ability of models trained with enriched data, as seen by their higher results. These findings demonstrate the importance of data augmentation in creating cyberbullying detection systems.

K-fold cross-valildation

Table 5 demonstrates the influence of different numbers of folds in k-fold cross-validation on the metrics of model performance. This analysis offers valuable information on how various fold configurations impact the accuracy of training and validation, as well as the metrics of precision, recall, and F1 score. As the number of folds grows from 1 to 5, there is a little variation noticed in the training and validation accuracy rates. The greatest accuracy reached is 97.88% for training and 96.82% for validation, both occurring with two folds. Moreover, the accuracy, recall, and F1-score metrics consistently maintain their values regardless of the fold settings, demonstrating the model’s consistency in performance when subjected to multiple cross-validation procedures. In summary, this approach helps in choosing the best fold configuration for k-fold cross-validation, which guarantees a reliable assessment of the model and an accurate estimate of its performance.

Analysis of model predictions on new text samples

Figure 8 shows both Bengali and English phrases were analyzed in the most current examination of the Fusion Transformer-XL model’s predictions on unseen text samples to comprehend the model’s decision-making process. For example, in the Bengali sentence Fig. 8A “আেরকটা বাংেরিজ পণ্িডত েবর হইেস পুরাই ফাউল,” “Another Bangree scholar came out full foul,” key terms like “আেরকটা,” “বাংেরিজ” and “পণ্িডত” played a crucial role in the text’s categorization by the model. Similarly, the sentence “েমেয় এক ইন্টারিভউতে বেলেছ েচােখ িবশ্বাস কেরনা অর্থাৎ নাস্িতক অিবশ্বােসর ধর্ম ইসলাম ধর্েমর েরাজা রােখ এটািক একটা”, “The girl said in an interview that she does not believe in the eyes, that is, the religion of atheistic disbelief fasts in the religion of Islam” featured critical words like “েমেয়”, “নাস্িতক,” and “ইসলাম,” impacting the prediction outcome.

In the English text depicted in Fig. 8B, words such as “zubear,” “real,” as well as “love,” “hurt,” and “asking,” were highlighted from phrases like “all love does is get you hurt and leave you asking questions” and “zubear says any real isn’t letting this happen.” These terms were crucial in determining how the model classified the text.

To sum up, this method of using LIME explanations makes it clear which particular words and phrases have the most effect on the model’s predictions. This approach facilitates the thorough evaluation of textual data for a range of categories, including trolling behaviour, religious context, and sexual content. Text categorization methods are more accurate and equitable when the influential terms in the sentences are highlighted. This makes it simpler to understand how the machine makes decisions.

Comparative analysis among and existing works

Table 6 presents our comparative study, where we analyze various classifiers employed for cyberbullying detection across different research studies. The applied classifiers range from traditional machine learning models such as SVM, RF, and LR to advanced deep learning architectures, including ConvNet, CapsNet, and transformer-based models. Notably, Teng & Varathan (2023) achieved a high accuracy of 97.41% using DistilBert, while Roy & Mali (2022) and Kumar & Sachdeva (2022b) reported strong performance with Transfer Learning and CapsNet, respectively. Similarly, Sihab-Us-Sakib et al. (2024) employed XLM-RoBERTa and Bangla-BERT, achieving an accuracy of 82.61%, while Diaz-Garcia & Carvalho (2025) conducted a comparative analysis of cyberbullying detection methods using BERT, RoBERTa, and GPT-2, with BERT achieving 97.34% accuracy. Our study introduces a novel approach that leverages a fusion of Transformer-XL models, achieving an exceptional accuracy of 98.17%. This fusion strategy not only surpasses existing methods in accuracy but also enhances model robustness, contributing significantly to the advancement of cyberbullying detection techniques.