Distilroberta2gnn: a new hybrid deep learning approach for aspect-based sentiment analysis

SemEval-2014 Task 4 Rest14

This dataset is derived from the SemEval 2014 challenge. It features an extensive compilation of restaurant reviews, methodically divided into training and testing subsets. The unique aspect of this dataset lies in its detailed annotation of sentences with one or more specific aspects of the restaurant experience, such as food quality, ambience, or service. Each aspect is evaluated for sentiment polarity, with annotations distinguishing between positive, neutral, or negative sentiments. This granularity allows for a nuanced analysis of customer opinions and preferences. This dataset provided a robust foundation for our analysis, with the training set comprising 2,164 positives, 633 neutral, and 805 negative annotations, totalling 3,602 instances. The testing set included 728 positive, 196 neutral, and 196 negative instances, culminating in 1,120 entries. This diverse assortment of sentiments offered a rich canvas for our sentiment analysis exploration.

SemEval-2015 Task 12 Rest15

Building on the foundations laid by Rest14, the Rest15 dataset from SemEval 2015 expands the horizon of sentiment analysis within the restaurant sector. It adheres to a similar structure, with its content organized into training and testing sets. Sentences within the reviews are tagged with aspects pertaining to the dining experience, accompanied by their respective sentiment polarities. The categorization into positive, neutral, and negative sentiments provides an invaluable resource for dissecting the multifaceted nature of customer feedback. For Rest15, the training set yielded 912 positive, 36 neutral, and 256 negative sentiments among 1,204 total instances. The testing dataset comprised 326 positive, 34 neutral, and 182 negative sentiments, aggregating 542 instances. The relatively smaller volume of neutral sentiments presented unique challenges and insights into sentiment distribution within this dataset.

SemEval-2016 Task 5 Rest16-EN

The Rest16-EN dataset continues the sequence emanating from SemEval 2016’s Task 5. It encapsulates a rich dataset of restaurant reviews, again segmented into training and testing sets, with each sentence annotated for specific aspects and their sentiment polarities. This dataset not only carries forward the tradition of its predecessors in providing a detailed look at the spectrum of customer sentiment but also introduces new challenges and opportunities for deeper sentiment analysis through its nuanced annotations. The dataset from Rest16-EN contributed 1,240 positive, 69 neutral, and 439 negative sentiments within the training set, amounting to 1,748 instances. The testing set accounted for 468 positive, 30 neutral, and 117 negative sentiments, leading to 615 instances. The nuanced distribution of sentiments across this dataset provided valuable perspectives on customer feedback trends.

SemEval-2016 Task 5 Rest16-ESP

The dataset Rest16-ESP is designed for Aspect-Based Sentiment Analysis in the restaurant domain. It contains annotated reviews in Spanish, where each review includes various aspects such as food, service, ambience, and price, each tagged with a sentiment that can be positive, negative, or neutral. The dataset aims to identify specific terms or phrases in the reviews that indicate discussed aspects and determine the sentiment expressed towards them, considering the context of the entire review or sentence. This dataset is valuable for developing and evaluating models automatically identifying and classifying aspects and their sentiments within the text, advancing sentiment analysis techniques for Spanish-language content in the restaurant domain. The dataset includes 1,925 positive, 120 neutral, and 674 negative sentiments within the training set, totalling 2,719 instances, and 750 positives, 48 neutral, and 274 negative sentiments in the testing set, totalling 1,072 instances.

Phase-1: text preprocessing techniques

In our approach to ABSA, we begin by meticulously preprocessing the text data to ensure it is primed for identifying and evaluating sentiments towards specific aspects within the text. Our first step involves the removal of HTML tags using a parsing library. This is essential for isolating the raw text from any HTML formatting, allowing us to focus exclusively on the textual content without distraction from web markup, which is extraneous to our sentiment analysis. As we progress through the preprocessing stages, our efforts are directed towards further refining the text to enhance its clarity and uniformity, thereby facilitating a more accurate sentiment analysis. We remove content enclosed within square brackets to eliminate any metadata, annotations, or extraneous information that does not contribute to the main sentiment or aspect being analyzed. Additionally, we meticulously cleanse the text of URLs, usernames, and other patterns that might introduce noise into our dataset, recognizing that such elements often detract from the meaningful sentiment analysis of specific aspects. Our preprocessing also includes removing punctuation and non-alphabetical characters, simplifying the text to its essential elements. This step is crucial for avoiding potential confusion caused by symbols or numbers, allowing us to focus on the words and their sentiments towards aspects. By converting all text to lowercase, we ensure uniformity, treating variations of a word, regardless of capitalization, as identical. This uniformity is vital for maintaining consistency in our analysis and enabling a more accurate evaluation of sentiments directed at specific features or attributes within the text.

These preprocessing steps are foundational to our work in ABSA. By preparing the text data carefully, we lay the groundwork for effectively discerning and assessing sentiments related to specific aspects. This preparation is not just about cleaning the data; it is about enhancing our ability to extract nuanced insights regarding how different elements or features within the text are perceived. Through this meticulous preparation, we aim to unlock deeper, more accurate understandings of sentiments within the text, thereby enriching our analysis and the insights we derive from it.

Phase 2: emphasize preprocessing

In our pursuit of refining ABSA, we have innovated a preprocessing technique that underscores the importance of aspect terms—key phrases within sentences representing the focus of sentiment analysis. By deliberately emphasizing these aspect terms—repeating them twice and enclosing them in special tokens ([ASP] and [/ASP])—we significantly augment the model’s capability to discern and assess sentiments related to specific aspects. This method, implemented before the transformer model processes the input, is designed to make aspect terms more salient, thereby influencing the attention mechanism to prioritize sentiment expressed about these aspects. The utility of this preprocessing technique lies in its ability to enhance the model’s aspect awareness and improve sentiment accuracy. It leverages the transformer model’s nuanced understanding of context, ensuring a sharper focus on aspect-specific sentiments. This approach facilitates more accurate sentiment analysis and contributes to the model’s training and inference efficiency by guiding its attention mechanism more effectively towards the aspects of interest without significantly increasing computational complexity. Through this specialized preprocessing step, we enrich the model’s input data in a particularly beneficial way for ABSA, showcasing our commitment to leveraging advancements in natural language processing for deeper, more insightful text analysis.

Phase 3: tokenization technique

The tokenization step is undertaken following the initial preprocessing phase, where aspect terms within sentences are emphasized to enhance aspect salience. This critical stage prepares our data for the transformer-based model. A pre-trained tokenizer is employed to convert the enhanced sentences from our training and validation datasets into sequences of tokens, which serve as the fundamental input units for the transformer model. To ensure uniformity and efficiency in processing, several critical parameters are specified during tokenization:

• max_length: This parameter ensures that all tokenized inputs conform to a uniform sequence length, denoted as seq_len. Sequences longer than seq_len are truncated, whereas shorter sequences are padded, standardizing input data length and facilitating batch processing.

• truncation: Enabled truncation ensures that inputs exceeding the predetermined maximum length (seq_len) are effectively shortened to fit within the model’s architectural constraints.

• padding: ‘Max_length’ padding is applied to ensure all sequences reach the required seq_len, enhancing consistency across the dataset and enabling efficient batch processing by the model.

• return_tensors = ‘tf’: This setting indicates that the output is formatted as TensorFlow tensors, aligning with the model’s framework and facilitating seamless integration into the training pipeline.

Applying these parameters during tokenization is meticulously designed to prepare the data in a manner that optimizes the transformer model’s ability to learn from and interpret the emphasized aspect terms. This step is pivotal in ensuring that the nuanced semantics encoded within the enhanced sentences are effectively captured and utilized during the model training and inference phases. Through this methodological rigor in data preparation, the importance of tailored preprocessing in NLP is underscored, particularly in enhancing the efficacy of sentiment analysis models. This approach not only optimizes the model’s performance but also paves the way for more sophisticated analyses in the field of ABSA.

Phase 4: constructing datasets for TensorFlow

In our research, preparing datasets for TensorFlow-based models was a crucial step, particularly in enhancing the efficiency and effectiveness of our ML project’s training and validation phases. The process is characterized by two pivotal stages: tokenization of the input texts and the structured assembly of TensorFlow datasets.

• Tokenization: The raw textual data was initially processed through tokenization, converting text into a machine-interpretable numerical format. This procedure yielded input_ids for the numerical representation of the texts and attention_mask to highlight the active parts of the data, optimizing the model’s focus during training.

• TensorFlow dataset assembly: Leveraging the tf.data.Dataset.from_tensor_slices function, we methodically constructed the training and validation datasets. This TensorFlow utility facilitated pairing tokenized inputs with their respective labels, creating a dataset format directly consumable by TensorFlow models. This approach streamlined the data preparation workflow and ensured that our neural network architectures optimally structured the input data for processing.

Incorporating the tf.data.Dataset.from_tensor_slices method into our dataset preparation process was instrumental in efficiently transforming raw textual data into structured formats suitable for model training. This methodology underscores the significance of precise data handling in the ML pipeline, reflecting our commitment to fostering model accuracy and performance through meticulous data preparation. Algorithm 1 illustrates the pre-processing technique of all our phases.

DistilBERT

In the comprehensive study presented by Sanh et al. (2019), the authors introduce DistilBERT, a model that addresses the computational inefficiencies and environmental concerns associated with the deployment of large-scale pre-trained models in NLP. Building upon the foundational work of BERT by Devlin et al. (2018), DistilBERT is proposed as a distilled version that retains the core linguistic capabilities of BERT while achieving substantial reductions in size and computational demand.

At the heart of DistilBERT’s development is the application of knowledge distillation, a technique where a smaller, more efficient model (the student) is trained to replicate the performance of a larger model (the teacher) Eq. (1).

(1) $$L_{\mathrm{C}\mathrm{E}}=-\sum _i{t}_i\log (s_i)$$where $t_i$ and $s_i$ denote the probability distributions predicted by the teacher and student models, respectively, for class $i$. The inclusion of this loss function allows DistilBERT to absorb nuanced probabilistic insights from BERT beyond mere label accuracy.

To further refine the distillation process, Sanh et al. (2019) introduce a triple loss function, which amalgamates the distillation loss $L_{CE}$, masked language modellingg loss $L_{MLM}$, and a cosine embedding loss $L_{COS}$. The latter aims to align the students’ and teachers’ hidden state vectors, fostering an intricate similarity in their representational spaces. This composite loss function plays a pivotal role in successfully distilling BERT’s linguistic intelligence into DistilBERT, facilitating the latter’s ability to achieve 97%. of BERT’s performance on benchmark tasks with only 60% of its original size.

Architecturally, DistilBERT mirrors BERT but with several strategic modifications aimed at efficiency. The model eschews token-type embeddings and the pooler and reduces the number of Transformer layers by half. These design choices are underscored by an intent to minimize computational overhead without disproportionately impacting the model’s performance—a empirically realised goal, as evidenced by DistilBERT’s performance on the GLUE benchmark.

The practical implications of DistilBERT’s design are profound. Through experimental validation, Sanh et al. (2019) demonstrate that DistilBERT not only maintains commendable accuracy across a suite of NLP tasks but also exhibits significantly enhanced inference speeds, particularly in on-device contexts such as mobile applications. This duality of performance and efficiency situates DistilBERT as an optimal choice for real-world applications where computational resources are at a premium.

DistilBERT has been established as a robust and efficient tool for sentiment analysis across various domains, demonstrating its prowess in processing and interpreting sentiments from textual data with notable accuracy and speed. Researchers, as seen in studies by Pramanik & Maliha (2022), Xiong & Yan (2021), have leveraged DistilBERT for general sentiment analysis tasks, where it has shown superior performance compared to traditional models like LSTM, especially in contexts requiring the analysis of product reviews and entity-level sentiments. These applications highlight DistilBERT’s capability to balance computational efficiency and the maintenance of a high degree of linguistic understanding, making it an attractive option for analyzing sentiments in large datasets.

However, to the author’s knowledge, DistilBERT has not yet been applied to the niche field of ABSA. ABSA, which requires identifying and evaluating sentiments towards specific aspects within a text, poses unique challenges that have not yet been explored with the application of DistilBERT. This gap presents an opportunity for future research to harness DistilBERT’s efficient architecture and adapt it to the nuanced demands of ABSA, potentially setting a new benchmark for performance in this specialized area of sentiment analysis.

GNN

Graphic neural networks (GNNs) represent a significant leap in the evolution of neural network architectures, enabling the effective processing of graph-structured data. This innovation traces back to the pioneering work by Scarselli et al. (2008), who introduced the foundational framework for GNNs, enabling direct application of neural networks to data represented as graphs. The primary motivation behind GNNs is their capability to capture the intricate patterns within data where entities are interrelated, making them particularly suited for complex systems such as social networks, biochemical structures, and communication networks. The operational principle of GNNs revolves around the concept of message passing or neighborhood aggregation, where the feature vector of each node is updated by aggregating features from its neighbors, thereby encapsulating both local and global structural information.

The mathematical formulation (of Eq. (2)) underpinning this process is expressed through the iterative update rule:

(2) $$h_v^{(t)}=\mathrm{U}\mathrm{P}\mathrm{D}\mathrm{A}\mathrm{T}\mathrm{E}\left(h_v^{(t-1)},\mathrm{A}\mathrm{G}\mathrm{G}\mathrm{R}\mathrm{E}\mathrm{G}\mathrm{A}\mathrm{T}\mathrm{E}\left(\{h_u^{(t-1)}:u\in \mathcal{N}(v)\}\right)\right),$$where $h_v^{(t)}$ denotes the feature vector of node $v$ at iteration $t$, and $\mathcal{N}(v)$ represents the neighbors of $v$. This formula underscores the mechanism through which nodes integrate and update information, facilitating a deep understanding of the graph structure.

The efficacy of GNNs extends across a myriad of applications, with recent research exploring their utility in nuanced fields such as ABSA. ABSA aims to discern the sentiment towards specific aspects within a text, necessitating an understanding of both the relational aspects within the data and the contextual sentiment, a challenge aptly addressed by the relational and contextual learning capabilities of GNNs. For instance, a recent study by Wang et al. (2022) implemented GNNs for ABSA to identify and analyze the sentiment directed at particular aspects of product reviews, demonstrating improved accuracy over traditional models. This reflects the growing interest and potential in applying GNNs to dissect and interpret complex, relationally structured sentiment data, paving the way for more insightful and granular sentiment analysis.

The advent and evolution of GNNs underscore a pivotal advancement in ML, offering a versatile and powerful tool for analyzing graph-structured data. Their application in fields like ABSA exemplifies their practical utility. It highlights the ongoing expansion of their applicative landscape, promising further innovations and applications in data science.

Architecture and building procedure of our DistilRoBERTa2GNN model

In this study, we first employed the DistilRoBERTa model. The architecture of DistilRoBERTa is depicted in Fig. 2. To capture the relative positions of tokens, input embedding and positional encoding were integrated as the model’s input. Multi-head attention mechanism computes the query (Q), key (K), and value (V) matrices by projecting the embedding dimension matrix of the sequence length. Subsequently, Q, K, and V are divided by the number of heads to process scaled dot-product attention (Fig. 3) parallelly.

Scaled dot-product attention, a variant of self-attention, employs scaling by $\sqrt{dk}$ to prevent the inner product from growing excessively, where $dk$ represents the vector dimension of the K matrix. The attention weights distribution, obtained by applying softmax to the dot product of Q and K, is divided by the scaling factor $1/\sqrt{dk}$. The resulting attention value is then computed by multiplying V with the attention weights distribution, as shown in Eq. (3).

(3) $$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q,K,V)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{Q{K}^T}{\sqrt{dk}}\right)V$$

The output of scaled dot-product attention, $h_1,h_2,h_3,\dots ,h_m$, is concatenated and multiplied with a weighted matrix $W_o$, yielding the multi-head attention output matrix (Eq. (4)). Multi-head attention enables the model to attend to information at multiple locations concurrently.

(4) $$\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}(Q,K,V)=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(h_1,h_2,h_3,\dots ,h_n)W_o$$

Following multi-head attention, a fully connected feed-forward network (FFN) process is executed (Eq. (5)). This process involves the linear transformation of the output matrix from the multi-head attention step, followed by the application of the Rectified Linear Unit (ReLU) function. Parameters W and $b$ are applied uniformly across positions, with variations introduced when transitioning across layers, hence referred to as position-wise transformation.

(5) $$\mathrm{F}\mathrm{F}\mathrm{N}(x)=\mathrm{M}\mathrm{a}\mathrm{x}(0,x{W}_1+b_1)W_2+b_2$$

In our research, we introduce the DistilRoBERTa2GNN model, a novel architecture that combines the strengths of pre-trained language models with the advanced capabilities of GNNs for sentiment classification. This model is specifically designed to leverage the rich, contextual embeddings generated by DistilRoBERTa, a distilled version of the robust RoBERTa model renowned for its efficiency and effectiveness in processing natural language.

The first phase of our approach involves utilizing DistilRoBERTa as a feature extraction mechanism. DistilRoBERTa, a lighter yet highly potent version of its predecessor, provides deep contextualized embeddings that capture the nuances and complexities of language in a compact form. By processing textual data through DistilRoBERTa, we obtain high-quality feature representations that encapsulate the input text’s semantic and syntactic characteristics. In doing so, we ensure that the intricate patterns and relationships within the text are preserved and made accessible for further analysis. Upon extracting these features, we strategically remove the output layer of DistilRoBERTa, shifting our focus from the pre-trained model’s original task to our specific objective of sentiment classification.

Following the feature extraction phase with DistilRoBERTa, our model transitions into the GNN component, where the extracted features are further processed and analyzed for sentiment classification. This seamless integration marks a pivotal step in our methodology, combining the nuanced textual understanding provided by DistilRoBERTa with the relational insight offered by GNNs. The GNN architecture is meticulously designed to handle the complexities of graph-structured data, beginning with the input layers that receive the contextual embeddings. These embeddings serve as the initial node features within the graph, representing individual pieces of text or entities within our dataset. To accommodate these features, our model employs reshaping layers, adjusting the dimensionality of the embeddings to fit the graph-based framework. This adjustment is crucial for aligning the data with the network’s structure, ensuring that each node is represented in a manner that preserves its contextual integrity while facilitating efficient processing. Central to our GNN framework are the dropout layers, strategically placed to enhance the model’s robustness against overfitting. By randomly omitting a subset of features during training, these layers encourage the model to develop more generalizable patterns, reducing its reliance on any single feature and improving its ability to perform sentiment classification across diverse datasets. The core of our GNN architecture is the mean aggregator layers, which play a vital role in synthesizing information across the graph. These layers aggregate features from the nodes’ neighbours, effectively capturing the graph’s local connectivity patterns and broader structural context. By averaging the features of neighbouring nodes, the mean aggregator layers ensure that each node’s representation is informed by its immediate context, enriching the model’s understanding of the relational dynamics at play.

Further processing and refinement of the aggregated features are achieved through additional reshaping and dropout layers. This hierarchical approach to feature processing streamlines the data and emphasizes the most salient features for sentiment classification. The culmination of this process is a transformation layer that prepares the features for the final classification task. This layer, often implemented through custom functions, adapts the processed graph features for specific outputs, enabling our model to accurately predict sentiment based on the complex interplay of textual content and relational context. At its core, the DistilRoBERTa2GNN model stands as a trailblazing synthesis of deep linguistic insight and graph-based exploration. By capitalizing on the rich, contextual embeddings from DistilRoBERTa and implementing a refined GNN framework to probe the relational dynamics of the data, our model introduces a nuanced and all-encompassing strategy for sentiment classification. This innovative structure not only propels the domain of sentiment analysis forward but also underscores the value of combining diverse AI techniques to solve intricate challenges in language understanding. In our proposed approach, we standardized our training approach by setting the batch size to eight, employing the Adam optimizer for its efficiency in handling sparse gradients and adaptive learning rates, and setting the learning rate to 0.001. We conducted our experiments over 50 epochs, allowing for sufficient model training to capture the complexity of sentiment classification while avoiding overfitting. This configuration strikes a balance between computational efficiency and model performance, providing a robust framework for our analysis.

Performance analysis of our model

The efficacy of our model is evaluated across these datasets, and the results are comprehensively detailed below. The experimental results demonstrate that the fusion of DistilRoBERTa with GNNs marks a significant advancement in the field of sentiment analysis, promising to enhance both the precision and depth of insights derived from text data. The evaluation metrics used to assess the model’s efficacy include the F1 score, precision, and recall, providing a comprehensive view of its performance across various scenarios. Table 3 illustrates the performance of our proposed approach with four datasets.

For the Rest14 dataset, the model achieved an F1 score of 77.98%, demonstrating robust performance. This metric, indicative of the model’s balanced precision and recall, reflects its effectiveness in harmonizing the true positive rate with the precision of its predictions. Specifically, a precision of 78.12% suggests that the model accurately identifies sentiments as positive or negative. A recall rate of 79.41% indicates the model’s efficiency in capturing relevant instances of sentiment within the dataset.

The analysis of the Rest15 dataset revealed an F1 score of 76.86%, showcasing a slight improvement in the balance between precision and recall compared to the Rest14 dataset. With a precision of 80.70% and a notably higher Recall of 79.37%, this increase in recall suggests an enhancement in the model’s ability to identify all relevant instances of sentiment, albeit at a slight expense of precision.

The Rest16-EN dataset highlighted the model’s superior performance, with an F1 score of 84.96%. This significant improvement reflects the model’s enhanced capability to identify and classify sentimental expressions accurately. The precision and recall were measured at 82.77% and 87.28%, respectively, highlighting the model’s exceptional accuracy and completeness in capturing sentiment expressions within this dataset.

For the Rest16-ESP dataset, a Spanish-language dataset, the DistilRoBERTa2GNN model achieved an F1 score of 74.87%, precision of 73.11%, and recall of 76.80%. These metrics underscore the model’s balanced performance in identifying and classifying sentiments accurately. The proposed model demonstrates a notable improvement in sentiment analysis tasks across various datasets, showcasing its robustness and effectiveness in handling complex sentiment expressions.

The experimental results from the evaluation of the proposed model across four distinct datasets demonstrate its robust and versatile performance in sentiment analysis tasks, encompassing both English and Spanish datasets. Characterized by high F1 scores and closely matched precision and recall rates, these results indicate the model’s efficacy in accurately and comprehensively identifying sentiments across various contexts. This study contributes valuable insights into the model’s applicability and effectiveness in sentiment analysis, showcasing its potential for wide-ranging applications in natural language processing and related fields.

The four figures (in Fig. 4) illustrate the performance of a proposed sentiment analysis model across four datasets using receiver operating characteristic (ROC) curves to assess its ability to classify different sentiment categories: “negative,” “neutral,” and “positive.” For Rest14, the model showed high accuracy in classifying “negative” (AUC: 0.93) and “positive” (AUC: 0.90) sentiments, with moderate accuracy for “neutral” sentiments (AUC: 0.80). In the Rest15 dataset, the model also performed well, particularly in identifying “negative” (AUC: 0.88) and “positive” (AUC: 0.87) sentiments, though it was less effective for “neutral” sentiments (AUC: 0.73). The Rest16-EN dataset revealed superior performance, with the model achieving an AUC of 0.94 for “negative” and 0.94 for “positive” sentiments, indicating a strong discriminative power. In contrast “neutral” sentiments were classified with good accuracy (AUC: 0.82). The current ROC curve demonstrates the model’s capacity to differentiate between sentiment categories with an AUC of 0.82 for “negative”, 0.67 for “neutral”, and 0.83 for “positive”, respectively, across the evaluated dataset. These curves significantly surpass the baseline performance of a random classifier, highlighted by a diagonal dashed line, underscoring the model’s effectiveness in sentiment analysis.