TurkSentGraphExp: an inherent graph aware explainability framework from pre-trained LLM for Turkish sentiment analysis

Explainability approaches for sentiment classification task in Turkish NLP domain

In the field of natural language processing in Turkish, various studies have been conducted regarding the classification of sentiments expressed in texts (Koksal & Ozgur, 2021; Ozcelik et al., 2021; Yıldırım et al., 2015; Dehkharghani, 2015). When the presented solutions are examined, it is observed that they roughly fall into three distinct categories. The first category involves studies that consider the structure and features of the language, perform syntactic and morphological analyses, and synthesize word stems extracted with the help of a sentiment lexicon or a lexical semantic network (WordNet; Ehsani, Solak & Yildiz, 2018) for emotion analysis. The second category comprises machine learning and deep learning-based models that rely on labeled data and are trained on a large number of manually labeled Turkish texts. Solutions created by fine-tuning pre-trained language models with a small amount of labeled data can also be included in this category. Finally, we can list solutions that represent texts on a graph network, trained using GRL where related words are interconnected, as solutions dependent on labeled data.

In studies belonging to the first category, the distinctiveness of the Turkish language both morphologically and syntactically from other languages is highlighted, and the importance of suffixes to the semantic meaning of the words is emphasized. Yıldırım et al. (2015) have studied the impact of preprocessing steps (tokenization, normalization, morphological analysis, and disambiguation) on classification accuracy for Turkish texts. The study was conducted on a Turkish social media dataset, and it was observed that normalization and morphological analysis in preprocessing increased sentiment accuracy by more than 5%. Support vector machines (SVM) was used as the classification model in this study. On the other hand, Eryiğit & Oflazer (2006) conducted a sentiment analysis study for Turkish texts using both dictionary and machine learning approaches (naive Bayes, support vector machines, and J48). Despite the complex morphological structure of Turkish, the use of a dictionary improved classification accuracy by 7%. Furthermore, in the studies Dehkharghani (2015) and Ozcelik et al. (2021), they developed Turkish sentiment lexicons for use in sentiment analysis of Turkish texts. In the SentiTurkNet by Dehkharghani (2015), a sentiment lexicon was created by manually annotating Turkish words found in the BalkaNet (Bilgin, Cetinoglu & Oflazer, 2004) the Turkish WordNet as positive, negative, or neutral. This lexicon allowed sentiment analysis at the text level by combining sentiment labels from words in Turkish texts using the lexicon. Due to the limited number of conceptual expressions in SentiTurkNet and the abundance of out-of-lexicon words, methods using this lexicon gained lower performance. A more comprehensive sentiment lexicon network called HisNet (Ozcelik et al., 2021) was developed by labeling the sentiment polarities of 76,000 words from another Turkish WordNet called KeNet (Ehsani, Solak & Yildiz, 2018). While dictionary-based methods have the advantage of being more expressive compared to machine learning-based methods, the building and maintenance of a sentiment lexicon, the inclusion of new concepts, and the exclusion of out-of-lexicon words have led to lower preference for them compared to modern sentiment analysis approaches.

In the study by Guven (2023), Turkish sentiment analysis was rigorously evaluated using a novel text-filtering method in conjunction with pre-trained language models. This innovative text filtering approach involves the exclusion of words within labeled texts that exhibit sentiments contrary to the assigned label. Pre-trained language models such as BERT (Devlin et al., 2018), A Lite BERT (ALBERT) (Lan et al., 2020), Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA) (Clark et al., 2020), and DistilBERT (Sanh et al., 2020) were fine-tuned and assessed on Turkish hotel and movie datasets. Notably, it was observed that when combined with the ELECTRA language model, the innovative text filtering method achieved state-of-the-art (SOTA) performance in Turkish sentiment analysis. On the other hand, Kilic & Buyukeke (2021) conducted a study on sentiment analysis of Turkish texts using an expressive approach known as Text-graph convolutional networks (GCN) (Yao, Mao & Luo, 2019), which is based on graph representation learning. To the best of our knowledge, there have been no prior instances in the literature of employing GRL for Turkish natural language processing tasks, especially in the context of sentiment analysis.

As it is evident, sentiment analysis studies conducted in Turkish language so far have predominantly relied on approaches that use word representations obtained independently from the context, disregarding the impact of compound words, phrases, affixes, and conjunctions on meaning. These approaches have been dependent on either a sentiment lexicon, a pre-trained language model, or an excessive amount of training data. In this research, we introduce a sentiment analysis framework tailored for the Turkish language. Our approach is rooted in natural language understanding and explainability. It takes into account context-sensitive words, phrases, and even linguistic affixes, considering the agglutinative nature of Turkish as a language. As a result, this framework allows for a profoundly interpretable sentiment analysis of Turkish texts.

Graph-based explainability approaches in NLP domain

Explainability in AI (XAI) powered systems and methods is crucial for transparency, fairness, trustworthiness, and improving overall reliability and usability. It plays a pivotal role in addressing ethical, regulatory, and practical challenges associated with the deployment of deep learning (DL) models. XAI employs specific techniques and methods to enable the traceability and explainability of each decision made during the machine learning (ML) process. On the other hand, AI often reaches a conclusion using a ML algorithm, but the developers of these systems cannot fully explain how the algorithm arrived at this conclusion. This complicates accuracy, limits transparency, and hinders accountability. XAI techniques consist of three main methods: consistency in predictions, traceability, and comprehensibility of the decisions made.

In particular, explainability presents a challenge in methods used for solving NLP problems. A three-step approach for XAI in NLP models is presented (Mishra, 2022). These steps involve word embeddings (input level), the inner workings of NLP models (processing level), and the models’ decisions (output level). One of the initial tasks at the word embedding level is to transform and visualize large-dimensional representation vectors into two or three dimensions. In this way, it becomes possible to observe how semantically related words cluster together. From this perspective, visualization at the representation level is seen as an effective XAI technique. An online approach to deriving an interpretable word embedding model (OIWE) (Luo et al., 2020) is one of the techniques developed in this direction. Furthermore, some approaches use external resources as XAI techniques at the embedding level. In these methods, representation vectors are created through a dictionary or a semantic lexical network, allowing semantically related words to cluster together (Faruqui et al., 2015; Orhan & Tulu, 2021).

At the processing level of XAI, there are two approaches: Post hoc interpretation and inherent interpretation. In the first approach, post hoc interpretation, efforts are made to dissect the hidden information in deep neural networks trained to solve an NLP problem syntactically and semantically. This allows us to observe the contributions of each input to both meaning and syntax. Techniques such as heatmap and t-SNE visualization can be used to visualize different semantic relationships between words in this context. On the other hand, inherently interpretable deep neural networks, such as recurrent neural networks (RNN) and long short term memories (LSTM), are trained in an explainable manner by adding transparency constraints and discovering tree and graph-like structures. Among the highly explainable inherently interpretable methods are the Sequential Iterative Soft-Thresholding Algorithm and deep unfolding (Hershey, Roux & Weninger, 2014; Wisdom et al., 2016) methods. In the case of the new-generation LLMs, transformers are used, and various studies have been conducted to understand the internal dynamics of transformers and visualize attention weights (Lee, Shin & Kim, 2017; Liu et al., 2018). ExBERT (Hoover, Strobelt & Gehrmann, 2019) is a visual tool designed specifically for the explainability of the BERT model. It creates explainability in the form of attention, compilation, and summary of text representations learned depending on the content by providing masking and proximity analysis. In the study by Wang et al. (2020), which focused on the GPT-2 model, a transformer-based large language model, they examined the relationship between syntactic dependency and attention weights and investigated tokens, Part of Speech (POS) tags, and head counts. They found that attention heads in the early layers focused on determiners, while in deeper layers, they focused on nouns. In the middle layers, attention was concentrated on dependency relations. As a result, they determined that heads in the early layers were more focused on position than content, while attention heads in the deeper layers were more focused on specific structures. Therefore, they concluded that long-distance relationships were captured through heads in the deeper layers. The general limitation of this approach is that attention does not have a unified definition of explainability. At the prediction level of models, approaches involving both post hoc and inherently interpretive methods have been adopted to enhance explainability. Post hoc interpretation approaches analyze the decisions made by a pre-trained model for a given input text. If the model’s architecture is not transparent, this approach is known as model-agnostic, while for models with transparent architectures, it is referred to as model-specific. In the study by Ribeiro, Singh & Guestrin (2016), a method called Local Interpretable Model-Agnostic Explanations (LIME) was proposed, which can be applied to the prediction layer of any machine learning model. Additionally, for NLP models at the prediction layer, XAI methods, such as perturbation and layer propagation, are also used (Samek et al., 2019). These methods are computationally costly since they do not require manual expert evaluation.

In the field of XAI, ongoing research explores various approaches, though standardized methods remain underdeveloped in ML, DL, and transfer learning. GRL has fewer XAI methods due to its novelty. Yuan et al. (2022) categorized graph neural networks (GNN) based XAI methods into two groups: instance-level (Funke, Khosla & Anand, 2020; Luo et al., 2020; Schlichtkrull, De Cao & Titov, 2020; Ying et al., 2019), which explain individual examples, and model-level (Yuan et al., 2020), which provide high-level insights. Their study applied these methods to sentiment analysis, constructing dependency-parsing-based graphs with words as nodes and relationships as edges. Node representations were initialized using BERT-generated vectors. GCN, graph attention networks (GAT), and graph isomorphism network (GIN) models were used for predictions, while words emphasizing sentiment were visualized for explainability. To evaluate explainability, the study used fidelity (masking key features to measure impact on accuracy), sparsity (highlighting influential features), stability (measuring accuracy consistency under small changes), and accuracy (comparing results with synthetic data). These metrics assessed the effectiveness of graph-based XAI methods for sentiment analysis in text data.

Graph construction module

Instead of representing review documents in the typical sequential manner, we model them in graph form. There are two primary reasons for this preference. Firstly, in the case of sequential dependencies between words in comments, long-range semantic relationships may be neglected. Using such graph representation, such long-range semantic relationships are preserved. Secondly, the preference arises from the realism of explanation being based on patterns of relationships between words rather than solely on the importance of individual words.

Constructing a Turkish text graph representation involves navigating several intricate challenges, particularly concerning the generation of context-aware token embeddings, handling of suffixes, and modeling dependency edges. Firstly, creating context-aware token embeddings demands a nuanced understanding of Turkish morphology and syntax to capture the rich contextual information embedded within the language. Given the agglutinative nature of Turkish, where suffixes can significantly alter the meaning of a word, accurately representing token embeddings requires careful consideration of the surrounding context. Additionally, effectively handling suffixes poses a considerable challenge, as they play a crucial role in Turkish word formation and can vary in form and function depending on the context. Managing these suffixes while maintaining the integrity of token representations is essential for constructing an accurate graph representation. Furthermore, modeling dependency edges to capture the relationships between words in Turkish text requires addressing the complexities of Turkish syntax, including flexible word order and complex syntactic structures.

The construction of a text representation graph requires addressing several challenges, including generating context-aware token embeddings, handling suffix variations, and modeling dependency edges in Turkish text. Leveraging a pre-trained language model provides an effective solution to these challenges due to its robust capability to capture rich contextual information and manage the complex morphological structure of Turkish. Pre-trained models are specifically advantageous because they are trained on vast amounts of diverse text data, enabling them to generalize effectively and encode linguistic nuances such as suffix-driven semantic changes and intricate syntactic dependencies. This makes them particularly well-suited for representing the agglutinative nature and linguistic complexity of Turkish text in graph-based formats.

Moreover, the bidirectional nature and fine-tuning capability of transformer-based language models enhance their ability to capture complex linguistic relationships, such as intricate semantic interactions and suffix-driven variations in Turkish text. This enables the effective representation of Turkish text in graph structures, preserving long-range dependencies and uncovering nuanced word relationships essential for accurate and explainable sentiment analysis (Dehkharghani, 2015; Htut et al., 2019; Limisiewicz, Rosa & Mareček, 2020).

The inference of context-aware word embeddings using a transformer-based pre-trained language model is outlined in “Context-Aware Word Embeddings”. The use of these vector values to address suffixes is described in “Handling of suffixes”, while “Edge Construction” explains the process of defining inter-word relationships and determining their weights.

Attentive classification module

In this module, a sentiment classification model for Turkish reviews is constructed based on the generated representation graphs.

This classification model is built using the GAT (Veličković et al., 2017; Brody, Alon & Yahav, 2021) architecture, which is a attention-based GRL model. The primary reason for this choice is the interpretability of the model’s decisions based on inherently learned attention values. In other words, it can be directly interpreted without constructing any ad-hoc model.

GAT uses attention to focus on informative neighboring nodes when updating a node’s embedding. However, a key limitation of GAT is its “static attention”: the importance of neighbors is determined solely by their features, independent of the specific node we are considering. GATv2 addresses this by modifying the computation order. This allows each node to attend to its neighbors dynamically, leading to a more expressive model capable of capturing complex graph structures compared to the original GAT.

From a technical standpoint, the solution to using this model needs to be formulated as transductive or inductive. The rationale behind this choice and the notational formulation are detailed in “Formulation of Problem: Inductive vs Transductive”. Then, in “Attention-based GNN Design for Sentiment Classification”, the construction of GAT model based on this formulation is introduced.

Explainability module

In this module, classification decisions are explained based on the attention values of the learned model. It essentially consists of two steps: in the first step, a weighted graph is constructed from the learned attention values, and then adaptive filtering is applied to these attention values to obtain a subgraph pattern as explainable purposes. This module is depicted on the right bottom side of Fig. 2.

Although GNN models have various approaches for explanation, they generally rely on ad-hoc proxy models, which can increase complexity and lead to information loss due to indirect explanations (Pope et al., 2019). To address this issue, in this module, we develop two different inherent explanation models based on the learned attention values from the last layer of the previous learning module. These models provide explanations at both the edge level and the subgraph level, aiming to overcome the complexity and information loss associated with ad-hoc proxy model approaches like GNNExplainer (Ying et al., 2019).

As described in “Attention-based GNN Design for Sentiment Classification”, the learning module’s last layer contains learned attention values for each edge based on the model’s decision. These values are used to create a weighted graph, with node values (words) included in the graph for consistency with the learning module. This graph is represented as $G_{\mathrm{a}\mathrm{t}\mathrm{t}}(V,E)$.

This module explains the decision-making process of the model, which is based on the attention values learned in the Attentive Classification module in “Attention-based GNN Design for Sentiment Classification”. This process develops an algorithm inspired by a adaptive background filtering method (Ostu, 1979), which is widely used in image processing problems.

This naive approach is to adaptively filter and rank the learned attention values. This solution algorithm is given in Algorithm 3.

The algorithm takes as input the trained model parameters/weights $\theta$ and the text representation graph $G(V,E)$, and it outputs the filtered explainability subgraph $G_{\mathrm{e}\mathrm{x}\mathrm{p}}$. The algorithm begins by computing edge attention weights $G_{\mathrm{a}\mathrm{t}\mathrm{t}}(V,E)$ using the model parameters $\theta$ and the input graph $G(V,E)$. It then proceeds to compute a histogram and a cumulative distribution function (CDF) from these attention weights to determine an adaptive threshold for filtering edges. The threshold is chosen based on maximizing the inter-class variance between two classes defined by the CDF. Finally, the algorithm filters the edges of the input graph G based on the computed threshold, resulting in the filtered explainability subgraph $G_{\mathrm{e}\mathrm{x}\mathrm{p}}$. This algorithm offers a systematic approach to extract informative edges from a graph, enhancing the interpretability and explainability of GNN models in text analysis tasks.

The time complexity of the proposed algorithm can be analyzed as follows. First, computing edge attention weights $G_{\mathrm{a}\mathrm{t}\mathrm{t}}(V,E)$ takes $O(E)$ time, where E is the number of edges in the input graph. Next, initializing and computing the histogram H and cumulative distribution function (CDF) C from the edge attention weights takes $O(E)$ time as well. The loop iterating through possible thresholds runs for $N-1$ iterations, where N is the number of bins in the histogram. Inside this loop, computing class probabilities, mean values, and inter-class variance for each threshold takes $O(1)$ time per iteration. Therefore, the total time complexity of the algorithm is $O(E+N)$, where E is the number of edges and N is the number of bins in the histogram.

As for space complexity, the algorithm requires space to store the input graph $G(V,E)$, the model parameters/weights $\theta$, the edge attention weights $G_{\mathrm{a}\mathrm{t}\mathrm{t}}(V,E)$, the histogram H and cumulative distribution function (CDF) C, and variables for computations such as mean values, class probabilities, and thresholds. The space complexity is dominated by the input graph and the edge attention weights, resulting in a space complexity of $O(E)$ for the input graph and $O(E)$ for the edge attention weights, leading to a total space complexity of $O(E)$ for the algorithm.

Benchmark datasets

The experiments have been performed on three real-world Turkish sentiment datasets with varying sizes. These are briefly described as follows:

• TripAdvisor: The dataset comprises 42,000 hotel reviews from the TripAdvisor web page (Büyükeke, Sökmen & Gencer, 2020), with sentiment labels in three categories. However, it is an unbalanced dataset, with 80% of the reviews being positive. To address this issue, we randomly selected 1,250 samples with an equal number of reviews in each class for this study.

• ImdbFilmReview: Film reviews from IMDB (Amasyalı et al., 2012), which contain three traditional sentiment categories: positive, negative and neutral.

• BlogPosts: Users’ sentiments from blog pages (Amasyalı et al., 2012).

The basic statistics of these datasets are listed in Table 1. Please note that the volume of $TripAdvisor$ dataset is much larger than the others. For this reason, this dataset was taken as a reference in some outputs given for demonstration purposes.

Furthermore, it is important to note that the BlogPosts dataset contains a higher number of classes due to its mood-based labeling approach, which differs from the traditional positive, negative, and neutral labels found in datasets like IMDB and TripAdvisor. The inclusion of this dataset is intended to evaluate the adaptability and robustness of the proposed solution in scenarios involving more nuanced label distributions. Specifically, the goal is to observe the performance of the GAT architecture, used as the backbone, compared to other architectures under default configurations across diverse class distributions. This allows for a comprehensive assessment of the model’s ability to handle complex or unconventional classification tasks, such as mood-based labeling.

Comparison models

As stated in “Graph Construction Module”, the framework proposes solutions for constructing graphs from Turkish texts, constructing an attentive model from these graphs, and explaining predictions. The main task is to build a successful model based on the graph representation. To evaluate the performance of the model, the two popular no-attention based GNN models were used:

• GCN: In GCNs (Kipf & Welling, 2016), a specialized model architecture is employed for tasks involving graph data, such as node classification and graph-level prediction. Unlike traditional neural networks, GCNs operate directly on graph structures, leveraging node features and their relationships. This approach allows GCNs to capture complex dependencies and patterns within graphs, making them effective for tasks like social network analysis, molecular structure prediction, and recommendation systems.

• GIN: Unlike GCNs that operate on fixed graph structures, GINs (Xu et al., 2018) dynamically consider different graph isomorphisms, enabling a more nuanced analysis of graph properties. This dynamic approach allows GINs to excel in tasks requiring precise graph matching and classification, such as molecular structure comparison, subgraph detection, and graph similarity assessment.

Please note that all of these models aim to learn graph representations and do not use the attention mechanism.

Evaluation metrics

For attentive classification module, macro F1-score (Hand & Christen, 2018) is preferred as evaluation metric due to several reasons. First, it addresses the challenge of class imbalance commonly observed in real-world graphs. Since some classes may have significantly more instances than others, the macro F1-score treats each class equally, preventing dominance by the majority class. Second, GNNs operate on local structural information within graphs, but global graph structure also matters. By encouraging a balanced consideration of all classes, macro F1-score promotes learning from both local and global contexts, improving model generalizability.

Technically, the F1-score combines precision and recall, providing a balanced measure for binary classification. For multi-class problems, we use variants like micro F1-score and macro F1-score. Here’s how they are defined:

Precision measures how many of the “positive” predictions made by the model were correct:

(9)

Recall measures how many of the positive class samples present in the dataset were correctly identified by the model:

(10)

The F1-score is the harmonic mean of precision and recall:

(11)

For multi-class problems, we compute the macro-averaged F1 score by taking a simple average of the class-wise F1 scores:

(12)

On the other hand, to empirically evaluate Explainability module, two commonly used metric is used. These are the fidelity and sparsity (Pope et al., 2019) metrics.

Fidelity and sparsity are two common metrics for evaluating the explainability of GNN. Fidelity measures how well the explanation preserves the prediction of the original model, while sparsity measures how concise the explanation is.

Formally, let $G=(V,E)$ be the original graph, and $G^{\prime }=(V^{\prime },E^{\prime })$ be the subgraph generated by the explanation method. Let $f(G)$ be the prediction of the GNN model on G, and $f(G^{\prime })$ be the prediction of the same model on G′.

Here, two different fidelity calculation approaches (Amara et al., 2022) are referred as follows and their harmonic mean is calculated as the fidelity value as shown in Eq. (15), where $w_{+}$ and $w_{-}$ are the weights for $Fi{d}_{+}$ and $Fi{d}_{-}$, respectively.

(13)

(14)

(15)

On the other hand, the calculation of the sparsity values of the generated explanations is calculated as in Eq. (16)

(16)

This metric also ranges from 0 to 1, where a higher value means a higher sparsity.

Implementation details

Our experimentation was conducted using the Python programming language, primarily selected for its open-source nature, robust community support, and extensive resource availability. Specifically, we leveraged the transformer library for experiments involving pre-trained language models and employed the pytorch-geometric library for experiments concerning GNN models. Additionally, authors developed their custom Python code to implement technical contributions such as Turkish text suffix normalization, graph construction, and adaptive attention filtering, as detailed in “Problem Statement”.

For our runtime environment, we utilized the Google Colab+ machine, which offers a Tesla K80 GPU, 12 GB of RAM, and an Intel Xeon CPU clocked at 2.20 GHz.

In our study, we utilized BERTurk as a pre-trained language model specifically tailored for Turkish text. This choice was motivated by its status as the most comprehensive transformer-based language model trained on Turkish data, providing a robust foundation for our experiments.

Results

Layer-wise embedding evaluation

As stated in “Graph Construction Module”, the graph construction process mainly rely on a pre-trained language model. To evaluate the quality of the vectors generated by the language model, as a response to RQ1, we conducted a visualization-based experiment on TripAdvisor dataset, the largest benchmark dataset. This experiment aims to demonstrate how the pre-trained language model embeddings capture semantic relationships without additional fine-tuning. Rather than training the language model on the dataset, this approach evaluates its effectiveness in inference mechanism, as described in “Context-Aware Word Embeddings”, showcasing its capability to capture semantic discrimination in sentiment analysis.

For the sake of simplicity, the document vectors are averaged naively over the word vectors and this process is done separately for each layer to obtain document embedding vectors.

Since the embedding vectors in the 12 layers of the model are 768-dimensional, we use T-SNE (Van der Maaten & Hinton, 2008) method to visualize them in two-dimensional space. Figure 3 depicts how the different sentiment classes are positioned in space for each layer. It is clearly observed that the documents are grouped more accurately as we move towards the higher layers. This confirms that the embedding vectors from the final layers provide the highest levels of semantic discrimination. These findings align with observations from previous studies conducted in other languages (Htut et al., 2019; Limisiewicz, Rosa & Mareček, 2020), validating that semantic discrimination in higher layers is also applicable to Turkish sentiment data, as outlined in RQ1.

Dependency graph accuracy

Unlike embedding vectors, attention matrices are located in each layer and in the heads of these layers. In order to evaluate these attention matrices, an accuracy metric is performed for each layer/head selection based on a labeled Dependency Graph dataset in the literature (Marşan et al., 2022). Experimental results are shown in Fig. 4. When we look at these results in general, it is observed that the accuracy values in the early layers are higher, that is, the syntactic relations are better modeled. This is in parallel with the experimental results of studies on other languages in the literature (Htut et al., 2019; Limisiewicz, Rosa & Mareček, 2020). In other words, this is the first time that the results of this experiment have been shown in the Turkish language. It is also observed that the selection of a single layer/head achieves max 0.2 success. This situation shows that more than one setting selection should be ensemble.

Please note that the search space for the optimal layer/head combination grows exponentially with the number of layers and heads as $2^{\mathrm{\#}layers\ast \mathrm{\#}heads}$. To avoid increasing computational complexity, we conduct experiments using overall average attention values.

Classification performance comparison

As explained in “Attentive Classification Module”, the $Classification$ module mainly relies on an attention based GNN model. To determine the most successful GNN model, a set of experiments are conducted over three real-world Turkish sentiment datasets. Additionally, a comparative analysis is carried out, considering the size of the data sets. The data sets are divided into those that use attention-based techniques and those that do not, in order to observe any differences in performance.

As detailed in “Proposed Approach”, TurkSentGraphExp, which integrates a graph-based learning mechanism to model inter-word relationships in sentiment analysis, underscores the importance of this structure for explainable sentiment analysis in Turkish texts. In this context, four existing graph-based models (GCN, GCI, GAT, GATv2) capable of learning from graph input representations were compared in terms of sentiment recognition, as shown in Table 2. Notably, the GAT variants (Veličković et al., 2017) GATv2 (Brody, Alon & Yahav, 2021), utilized as the backbone of TurkSentGraphExp, outperformed the other models with margins of 0.1, 0.8, and 0.53, respectively. These results highlight the effectiveness of attention-based GRL models in recognizing sentiment.

Table 2:

Model comparison with macro-F1 scores on benchmark datasets.

Attn	Model	TrpAdvisor	Imdb	BlgPost
No	GCN	0.11	0.50	0.30
No	GIN	0.21	0.16	0.31
Yes	Ours/wGAT	0.74	0.48	0.32
Yes	Ours/w $GATv2$	0.70	0.58	0.32

DOI: 10.7717/peerj-cs.2729/table-2

Note:

Values in bold denote that the GAT variants outperformed the other models with margins of 0.1, 0.8, and 0.53, respectively.

Additionally, it is observed that even the most successful model on $ImdbFilmReview$ and $BlogPosts$ data, where the number of documents is low, remains at 40–50% levels. However, when the size of the dataset in $TripAdvisor$ data is large, it is observed that the success of the model is quite high. This observation indicates that the size of the training set may affect the performance of the model.

Finally, in order to ascertain whether the learning process of the model is exhibiting signs of overfitting, the loss values are presented in Fig. 5. Upon examination of these values, it becomes evident that there is no discernible difference in loss between the training and validation data. This observation indicates that the learning model is not exhibiting a tendency towards overfitting.

Evaluation of explainability

To answer RQ4, the explainability of the proposed model is evaluated in qualitatively and quantitatively. For the qualitative evaluation, observations are made through some specific extreme case studies, while for the quantitative evaluation the metrics described in “Evaluation Metrics” are used. Specifically, for the qualitative evaluation, we focused on situations with different emotional expressions such as neutral or mixed sentiment reviews. This is because traditional approaches modelled without the use of graphs/dependencies fail to classify texts in this form (Demirci, Keskin & Doğan, 2019; Yıldırım et al., 2015). For example, let’s consider the following comments containing three neutral or mixed emotions for demo purposes:

• Otel (The hotel) konforlu (comfortable), ancak (but) hizmet (service) yetersiz (insufficient).

• Oda (The room) geniş (spacious), fakat (yet) temizlik (cleanliness) sorunlu (problematic).

• Personel (Staff) yardımsever (helpful), lakin (however) yemekler (meals) lezzetsiz(tasteless).

The explanations of these three interpretations derived from the sentiment classification predictions are shown in Figs. 6–8. It is observed that the sentiment phrases are detected with high weights by the proposed framework, while the overall discriminability of the explanations produced by the GNNExplainer model is low.

From a quantitative perspective, two common approaches are used in the literature to measure the explanation quality of models: fidelity and sparsity. By applying these approaches as formulated in “Evaluation Metrics”, the explanation quality of the proposed model is observed in Fig. 9. With this observation, the results are compared with a popular GNN explainability model called GNNExplainer. According to these results, the higher fidelity value of the proposed framework according to the changing sparsity rate indicates that more faithful explanations are produced in terms of the quantity of explanations. On the other hand, since the proposed model works inherently, there is no need for an ad-hoc model, it is obvious that the model produces faster results with less complexity. Such quantitative observations show that the proposed model is effective.