A syntax-injected approach for faster and more accurate sentiment analysis

Introduction

Sentiment Analysis (SA) holds a pivotal role in Natural Language Processing (NLP), dealing with the identification of subjective assessments, such as opinions on hotels in textual content (subjectivity identification task), and predicting the polarity of these assessments, whether they are positive or negative (Polarity Classification task, PC), as exemplified in (1) (Cui et al., 2023).¹

(1) Anne: This hotel is awesome. (PC: Positive) Ben: This hotel is terrible. (PC: Negative)

Some approaches to SA rely on identifying sentiment-bearing words in a sentence, such as “awesome” and “terrible” in (1), and using them as key indicators of polarity.

These are known as sentiment words, and are listed together with their associated polarities in sentiment dictionaries like SO-CAL (Brooke, Tofiloski & Taboada, 2009). Lexicon-based methods like this offer an intuitive, transparent, and easily explainable analysis, as the sentiment polarity associated with a sentence can be traced back to the specific words that triggered it. This stands in contrast to supervised machine learning approaches, which rely on black-box models that infer predictions from training data but do not offer interpretability or word-level traceability (Cui et al., 2023).

However, the polarity of a sentiment word can be shifted by various linguistic elements, notably including negation, as in “This hotel is not good.” As a consequence, if high accuracy is desired, dictionary-based approaches cannot simply rely on sentiment dictionaries like SO-CAL to perform SA tasks. These approaches should also somehow model the relation between sentiment words and polarity shifting elements like negation. Since the scope of relevant phenomena such as negation can be determined from sentence syntax, syntactic parsing is a natural way to deal with the syntactic relations between sentiment words and polarity-shifting constructs. This is achieved by applying rules that predict sentence polarity based on the structure of a parse tree. These rules offer an explainable and transparent approach and tend to be more accurate than simple local heuristics, such as assigning negation scope to a fixed-length window of tokens (Thelwall et al., 2010). The most widely used formalism for syntactic parsing is Universal Dependencies (UD) (Zeman et al., 2023). UD is a dependency-based framework that represents the syntactic structure of a sentence as a parse tree composed of directed binary relations (called dependencies) between words. These dependencies specify the syntactic roles that words play in the sentence, such as subject, direct object, or adverbial modifier.

Stanza is an example of a popular NLP tool that provides syntactic dependency parsing in the UD formalism in several languages, including Spanish and English, our focus languages in this article (Qi et al., 2020). It has been shown that approaches that use syntactic parsing for an explainable and transparent SA based on the UD formalism or predecessor versions (Vilares, Gómez-Rodríguez & Alonso, 2017; Kellert et al., 2023) show better performance in polarity prediction tasks than heuristic approaches that do not rely on syntactic parsing at all such as VADER (Hutto & Gilbert, 2014), as demonstrated by Gómez-Rodríguez, Alonso-Alonso & Vilares (2019).

Despite the clear advantages of using syntactic parsing for SA, conventional syntactic parsers like Stanza (Qi et al., 2020) are often slow and computationally costly (Gómez-Rodríguez & Vilares, 2018; Roca, Vilares & Gómez-Rodríguez, 2023; Strzyz, Vilares & Gómez-Rodríguez, 2019b). Several solutions have been suggested in recent years to address this problem by developing faster systems by the transformation of dependency parsing into a SEquence Labeling (SEL) task (Gómez-Rodríguez & Vilares, 2018; Roca, Vilares & Gómez-Rodríguez, 2023; Strzyz, Vilares & Gómez-Rodríguez, 2019b). However, so far, SEL parsers have not been applied to SA. This article fills this gap. Thus, we adopt the SEL dependency parsing approach for sentiment analysis, to speed up the application of syntactic rules for polarity prediction and evaluate its practicality. In addition, we test our approach with several dictionaries to see which one yields the highest accuracy when used with the SEL-based system.

Our contributions are: (1) We present, to the best of our knowledge, the first approach that performs fast, syntax-based polarity classification using sequence labeling parsing. For this purpose, we first train an SEL system to perform syntactic dependency parsing (SELSP) in English and Spanish. After measuring its parsing accuracy on test sets of these languages, we then apply our SESLP in the domain of SA for the polarity prediction task through a sentiment analysis engine which relies on syntax-based rules to calculate polarity in English and Spanish. (2) We compare the performance of our approach in terms of runtime and accuracy in both languages using the same syntactic rules on top of the well-known dependency parser from Stanza, as well as to VADER—a heuristic approach to SA that does not use syntax trees and is widely used in industry—and to a state-of-the-art Transformer-based (RoBERTa) model. (3) We study the influence of sentiment dictionaries and their interaction with syntax-based systems by using several sentiment dictionaries, and combinations thereof, on top of the same set of rules and the two mentioned parsers.

Related work

The most widely used approaches to perform SA include Machine Learning/Deep Learning (ML/DL) methods, lexicon-based methods, and hybrid methods.

Syntactic parsing for sentiment analysis

Syntactic parsing is the task of determining the hierarchical structure of a sentence in natural language. For this purpose, two main kinds of representations are used in NLP: constituent parsing, where the sentence is recursively divided into smaller units, and dependency parsing, which focuses on representing relations between words. While constituent parsing has seen some use in sentiment analysis (Socher et al., 2013), dependency parsing is the dominant alternative for the task, as it straightforwardly represents what words are modified by other words. In particular, it is the basis of all the methods based on syntactic rules cited in the previous section.

In the last few years, Universal Dependencies (UD) (Zeman et al., 2023) has become the predominant framework for dependency parsing, especially in multilingual setups, as it provides harmonized annotation guidelines and freely-available corpora for over 100 diverse languages.

Syntactic parsing within UD involves the analysis of the grammatical structure of sentences to determine the roles of each of their parts and the relationships between them. It includes text tokenization that breaks down the input sentence into individual words or tokens, Part-of-Speech (POS) tagging where each token is assigned a tag indicating its grammatical category (e.g., noun, verb, adjective), and finally dependency parsing itself, where the syntactic relationships between words in a sentence are determined. Like all dependency grammar frameworks, UD represents syntactic relationships as directed links between words, where one word is the head and the other is the dependent. The dependency relation typically captures grammatical relations such as subject, object, modifier, etc. In UD, the syntactic relations are expressed in a standardized framework for annotating dependency trees for different languages. The output of the syntactic parsing in UD is a dependency tree, which is a hierarchical representation of the syntactic structure of the sentence. Each node in the tree represents a word, and the edges capture the syntactic relationships between words.

Kellert et al. (2023) make use of UD-based syntactic parsing and exploit dependency relations expressed in UD between sentiment words and other words that can shift, strengthen, or weaken the polarity of sentiment words (see also Vilares, Gómez-Rodríguez & Alonso (2017) for application of syntactic parsing for SA using a predecessor framework to UD). The first stage of the approach in Kellert et al. (2023) is to search for sentiment words taken from a sentiment dictionary SO-CAL (Brooke, Tofiloski & Taboada, 2009) in the input text, such as the sentiment word wonderful (in Fig. 1). Then, the system traverses the dependency tree and applies a set of syntax-based rules to check whether negation words or other modifiers and relevant syntactic phenomena change the polarity of sentiment words (see the dependency tree in Fig. 1). This syntactic approach applies recursive and compositional rules to negation and intensification, to identify the right scope of polarity shifting elements and to properly compute the sentence polarity (Vilares, Gómez-Rodríguez & Alonso, 2017; Kellert et al., 2023). This means that the computing of the sentence polarity is done bottom-up: first, intensifiers and polarity-shifting syntactic phenomena in the structurally lowest levels of the tree are used to modify the polarity of their modified sentiment-bearing words and expressions. Then, structurally higher polarity shifting elements are computed, and polarities from converging branches of the dependency tree are aggregated, until the root of the sentence structure is reached and the final combined polarity for the whole sentence is obtained (Vilares, Gómez-Rodríguez & Alonso, 2017; Kellert et al., 2023). For instance, in the sentence in Fig. 1, first, the modification of the adjective “kind” by the adverb “very” is computed, and then the conjunction between the sentiment word “wonderful” and the output of “very kind” is computed. Since there are no further polarity-shifting elements, the polarity of this conjunction is output as sentence polarity.

Dependency parsing approaches

To successfully use UD-based dependency parsing for sentiment analysis as described in the previous subsection, one ideally would want the parser to be accurate, so the obtained trees provide reliable information to compute polarity, and efficient, so that the system can operate at a large scale with a reasonable resource consumption. However, choosing a parsing approach involves a tradeoff between speed and accuracy, as the most accurate systems tend to be slower (Anderson & Gómez-Rodríguez, 2021). In this respect, it is worth noting that Gómez-Rodríguez, Alonso-Alonso & Vilares (2019) showed that the polarity classification task has modest accuracy requirements, i.e., there is a “good enough” parsing accuracy threshold past which increasing accuracy doesn’t bring further improvements to the end task. On the other hand, speed is often a practical bottleneck, especially if a large number of messages need to be processed.

However, as already noted in § Introduction, conventional syntactic parsers based on ad-hoc algorithms, like the one included in the Stanza library (Qi et al., 2020) are often slow and computationally costly (Gómez-Rodríguez & Vilares, 2018; Roca, Vilares & Gómez-Rodríguez, 2023; Strzyz, Vilares & Gómez-Rodríguez, 2019b). In recent years, however, an alternative approach has emerged that offers significantly faster parsing, based on viewing dependency parsing as a sequence labeling task. Sequence labeling can be performed in linear time with respect to the length of the input, and is highly parallelizable, so it provides high practical speeds (Strzyz, Vilares & Gómez-Rodríguez, 2019b; Anderson & Gómez-Rodríguez, 2021).

Previous work on syntactic parsing as sequence labeling has been applied to several NLP tasks such as integration of constituency and dependency parsing (Strzyz, Vilares & Gómez-Rodríguez, 2019a), semantic role labeling (Wang et al., 2019) or LLM probing (Vilares et al., 2020), where its speed and simplicity can be useful. CoDeLin (COnstituent and DEpendency LINearization system, Roca, Vilares & Gómez-Rodríguez, 2023) is a recent comprehensive library for parsing as sequence labeling that supports both constituent and dependency parsing. The basic idea behind dependency parsing in CoDeLin is to assign to each word a tag with information about a part of the dependency tree involving that word, in such a way that a full tree can be decoded from the sequence of labels for a given sentence. For this purpose, there are several types of encoding functions, each of which corresponds to a different way of representing the dependencies between words in a sentence in the form of tags associated to each of the words. We now summarize the framework and some of the encodings of interest. More information, including further encoding functions, can be found in Strzyz, Vilares & Gómez-Rodríguez (2019b), Roca, Vilares & Gómez-Rodríguez (2023), and Gómez-Rodríguez, Roca & Vilares (2023).

Following Roca, Vilares & Gómez-Rodríguez (2023), let $T_{|w|}$ be the set of dependency trees for sentences of length $|w|$. Each element of $T_{|w|}$ is a dependency tree $t_w=(V,A)$ for a sentence $w=[w_1,\dots ,w_{|w|}]$ where each $i\in V$ denotes the position of the word $w_i$, and A is the set of dependency relations between words, where each $a\in A$ is of the form $a=(h,r,d)$ such that $h\in V$ is the index of the head of the relation, $d\in V$ corresponds to the index of the dependent, and $r\in R$ is a label describing a syntactic function, i.e., the type of syntactic relation between head and dependent (such as subject, modifier, indirect object, etc.). We can cast the dependency parsing task as a sequence labeling task by defining a set of labels L that will let us encode each tree $t_w\in T_{|w|}$ as a sequence of $|w|$ labels $l_w\in L^{|w|}$, and an encoding function $F_{|w|}:T_{|w|}\to L^{|w|}$ that allows us to translate the trees into those labels. $F_{|w|}$ must be injective, so that each tree is encoded as a unique label sequence and correct label sequences can be decoded back into trees. While injectivity is not enough to guarantee that incorrect label sequences that might be output by a machine learning sequence-labeling system can also be decoded (that would require surjectivity, which known encoding functions do not have), such sequences are handled in practice with straightforward heuristics, like ignoring labels that are contradictory or violate tree constraints.

In all encoding functions that we will consider here, the $j$th label in the sequence $l_w$ assigned to a sentence $w$ is a tuple $l_j=(x_j,r_j)$, where $r_j$ is always the relation type (syntactic function) describing the dependency from $w_j$’s parent to $w_j$; whereas $x_j$ is an encoding-specific value that encodes the location of the head of $w_j$.² In particular, we consider three encodings, where $x_j$ is defined as follows:

• Absolute encoding: $x_j$ directly represents the absolute position of the head of $w_j$, i.e., $x_j=i\mid (i,r,j)\in A$.

• Relative encoding: $x_j$ represents the position of the head of $w_j$ as an offset or displacement of the head with respect to the dependent, i.e., $x_j=i-j\mid (i,r,j)\in A$. Thus, a positive value $+x$ means that the head is $x$ positions to the right of $w_j$, whereas a negative value $-x$ means that it is $x$ positions to the left.

• POS-encoding: $x_j$ is in turn a tuple $(p_j,o_j)$, where $p_j$ is the part-of-speech tag of the head of $w_j$, and $o_j$ is a displacement where only words tagged $p_j$ are taken into account. Thus, a label $(p,+x)$ for a positive value $x$ means that the head is the $x$th among the words tagged $p$ that are to the right of $w_j$, while $(p,-x)$ means that it is the $x$th word tagged $p$ to the left of $w_j$.

Figure 2 illustrates how a sentence’s parse tree is represented in terms of different encodings in CoDeLin (Strzyz, Vilares & Gómez-Rodríguez, 2019b; Roca, Vilares & Gómez-Rodríguez, 2023). Once one has chosen a tree encoding, training a parser is straightforward: a sequence-labeling model is trained to label sentences with the syntactic labels in the selected encoding, and at test time, the generated labels are decoded back to trees by using the displacements (and PoS tags, in the PoS-based encoding) to find the head of each node and build the corresponding dependency. If the model outputs a wrong label that causes a contradiction or constraint violation (e.g., a head offset pointing to a non-existing word position like position 6 in a sentence of length 4, or causing a cycle), that label is simply ignored and no dependency is created.

Until now, parsing as SEL (as provided e.g., in CoDeLIn) has not been applied yet to the domain of SA using SEL-based dependency models. In this study, we fill this gap in previous research, to see if this fast parsing approach can improve the speed of syntactic rule-based sentiment analysis, and at what cost (if any) to its accuracy.

Materials and Methods

Methodology

The sentiment analysis methodology consists of two modules; a sequence labeling-based dependency parser (SELSP) to generate dependency trees followed by a sentiment analysis engine which uses sentiment dictionaries and syntactic rules for sentiment polarity classification from the said dependency trees as shown in Fig. 3.

Syntactic parsing (SELSP Parser) State-of-the-art syntactic parsers (Kondratyuk & Straka, 2019; Mrini et al., 2019) nowadays mostly use Transformer architectures due to their ability to capture complex linguistic patterns and dependencies efficiently. We specifically chose DistilBERT—a distilled version of BERT—to balance performance with computational efficiency. Unlike lightweight models (e.g., MobileBERT, TinyBERT), DistilBERT (Sanh et al., 2019) preserves 95% of BERT’s performance while being 40% smaller and 60% faster. It requires less computational resources and suits well for the tasks where both accuracy and speed are crucial, such as dependency parsing. Benchmark studies (Sanh et al., 2019) confirm its competitive accuracy on GLUE and SQuAD, with marginal degradation compared to BERT.

To train the SELSP, we used DistilBERT-Base-Uncased as our sequence labeling classifier and evaluated the three sequence-labeling encodings mentioned in Section “Dependency parsing approaches” on the validation sets of the UD EWT English dataset (Silveira et al., 2014) and the UD Spanish AnCora (Taulé, Martí & Recasens, 2008) dataset. We chose the Relative Encoding (REL) as shown in Fig. 2, as it obtained the highest accuracy in these experiments.

We fine-tuned the DistilBERT model (Sanh et al., 2019) from the pretrained checkpoint (DistilBERT-base-uncased) available in the HuggingFace library using the UD-EWT English dataset and the UD Spanish AnCora dataset. DistilBERT consists of a pre-trained transformer encoder, and during fine-tuning, only the parameters of this encoder are updated. The basic idea behind fine-tuning was to leverage the knowledge learned by DistilBERT during pre-training on a large corpus and to adapt it for the sequence labeling task.

To fine-tune the SELSP, the original training, validation and test data splits from the considered English and Spanish UD datasets were first converted into sequence labels using the CoDeLin framework (Dependency Parsing Approaches). Then, the model was trained for 30 epochs on the resulting training set using the Hugging Face Trainer framework, with validation performed after each epoch to adjust the learned weights and biases. Training was conducted on an NVIDIA GeForce RTX 3090 GPU with a fixed random seed (seed = 42) for reproducibility. Table 3 describes the other hyperparameters used to train the model. The same hyperparameters were used to train both English and Spanish models. The pipeline of SELSP parser is given in Fig. 4.

The training process results into a model that, given an input sentence, predicts the labels encoding the dependency heads and relations in our chosen encoding, in this case the Relative Encoding (REL). We then use the CoDeLin sequence labeling framework to decode these rules produced by the SELSP into the CoNLL-U format that is standard for Universal Dependencies. With this, the accuracy of the parse trees produced by the fine-tuned SELSP was evaluated using the CoNLL 2018 standard evaluation script (Zeman et al., 2018).

The CoNLL-UD format represents a dependency graph in a particular format according to which tokens are annotated linguistically concerning their lemma, POS, morphology, the location of their syntactic head expressed as an absolute token index, and the label of the dependency relation such as subject relation (nsubj), object relation (obj), determiner (det), adverbial modification (advmod), conjunction (cc), conjoined element (conj), complementizer (xcomp), or punctuation (punct), as shown in Fig. 1. In this format, word tokens are indexed with integers starting from 1, and all the information about a given token is in a separate line. The information of each dependency relation is included in the line associated with the dependent, by specifying the index of its head together with the relation type. For instance, in the example in the figure, the head of the word token with the integer 2 is the word 3 (the main verb), with relation type “nsubj” indicating that “I” is the subject of “found”; and the modifier “very” at position 10 is modifying the word “kind” at position 11. The sentence’s syntactic root (the main verb “found” in the example) has no head, so its associated head index is set to zero. This information generated by the dependency parser is used to identify polarity shifters: for example, since “very” is a modifier of “kind”, it will increase the positive polarity associated with “kind”. The polarity prediction demonstrated between the dependency graph and the CoNLL-UD format in Fig. 1 will be explained below.

Sentiment analysis engine for polarity classification Once the SELSP is trained, its output (dependency tree) is fed into the sentiment analysis engine which uses syntactic rules (Kellert et al., 2023) to calculate sentiment polarity from said dependency trees. The overall architecture of the sentiment analysis system is provided in Fig. 3.

The original sentiment analysis system of Kellert et al. (2023) contains information on how to calculate polarity scores of reviews based on sentiment words from the sentiment dictionary SO-CAL (Brooke, Tofiloski & Taboada, 2009) and UD-based syntactic rules for Spanish, using syntactic parse trees obtained from Stanza (Kellert et al., 2023). Apart from incorporating the option of parsing with SELSP as described above, we also extended the system to handle both English and Spanish, as well as to be able to incorporate different sentiment dictionaries apart from SO-CAL. We now describe the extensions made.

We use a set of English negation words and intensifiers like “very” or weakeners like “few” to process English sentiment data, which can be found in “Appendix B”. The set of negation words we use for processing Spanish data is integrated into the code from Kellert et al. (2023).

We use several English and Spanish sentiment dictionaries. For the English sentiment dataset, we use the following dictionaries: (1) SO-CAL, which is available in Spanish and English (Brooke, Tofiloski & Taboada, 2009), (2) VADER Lexicon (VADERdict), which is only available in English (Hutto & Gilbert, 2014), (3) an aggregate version that we obtain by adding to SO-CAL the sentiment words from VADERdict that are not initially covered by SO-CAL, and (4) the English dictionary from BabelSenticNet (Vilares et al., 2018). For the two Spanish sentiment datasets (OpeNER reviews and the 10,000 Spanish TripAdvisor reviews from RestMex), we use (1) the Spanish version of SO-CAL, (2) an aggregate version that combines SO-CAL with words from VADERdict that are missing in the English SO-CAL that we translate into Spanish via DeepL (SO-CAL + sp. VADERdict), and two automatically-extracted datasets: (3) words with low ambiguity extracted from titles from the Rest-Mex training dataset (Kellert, Gómez-Rodríguez & Uz Zaman, 2024) and (4) an aggregate version that combines SO-CAL with words from VADERdict and Rest-Mex dictionary.

We now provide some details on how these two latter dictionaries have been generated. We measure the polarity ambiguity of words by the standard deviation of polarity judgments (sdv) (Kellert, Gómez-Rodríguez & Uz Zaman, 2024). The authors of said article motivate their method of constructing sentiment dictionaries by using words with low sdv from titles as follows. Titles of reviews represent short summaries of long reviews and the words from these summaries often function like sentiment words, e.g., “Recommendable!”, “Excellent!”, etc. However, sometimes titles also represent topics of reviews, like “Last holidays”, and the words from these topic-titles cannot be considered as sentiment words. In this case, as the authors show (Kellert, Gómez-Rodríguez & Uz Zaman, 2024), words from these titles tend to be associated with a high sdv, since people will provide different, sometimes opposite judgments, to reviews containing that kind of topic words. For this reason, the authors consider only words with low sdv as good candidates for building sentiment words out of titles. Thus, we only select words with lower polarity variation measured by sdv (<0.6). For the experiments in this study, we use two different word lists. One word list contains words with low sdv from short titles (ShortT), which are those that contain only one word such as “Excellent!”, and the other list contains words with low sdv from short and long titles (ShortT + LongT), where long titles are those with multiple words. It is important to note that the accuracy calculation of the polarity prediction task based on the Rest-Mex dataset was performed on whole review texts excluding their titles. As a consequence, the sentiment dictionary created from the titles of the Rest-Mex dataset does not influence the accuracy of the polarity prediction task based on reviews from the Rest-Mex dataset.

Regarding aggregation of sentiment dictionaries (like when we combine SO-CAL with VADERdict), the issue can arise that two dictionaries have different polarities for the same word. Throughout the article, we will consider that SO-CAL is the basic dictionary to which we add words with corresponding polarities from other dictionaries that are not contained in SO-CAL. Thus, if SO-CAL has a lemma/word with a given polarity, but we combine it with another dictionary (such as VADERdict or the dictionaries extracted from Rest-Mex data) where the same lemma/word has a different polarity, we always keep the score assigned by SO-CAL.

For all these sentiment dictionaries, we compare the polarity results produced by our system that combines SELSP and the sentiment analysis engine with two main baselines. The first one is the version of the same SA rules that used Stanza-based syntactic parsing as their backbone instead (Kellert et al., 2023). The second main baseline is the shallow heuristics-based approach VADER, which is a widely used in practical applications of SA in research and industry use cases (see e.g., Lasser et al., 2023; Deas et al., 2023; Yuliansyah et al., 2024; Barik & Misra, 2024; Lahande, Lahande & Kaveri, 2025 for some recent examples).

We also compare our results from the Spanish polarity prediction task with the state-of-the-art Transformer-based model from the javier-alonso team that achieved the second rank in the polarity prediction task in the Rest-Mex shared task 2023 (Álvarez-Carmona et al., 2023; Alonso-Mencía, 2023). The javier-alonso team (Alonso-Mencía, 2023) used two variants of the RoBERTa model, namely roberta-base-bne and twitter-xlm-roberta-base. They balanced the effect of minority classes (such as polarity 1 and 2) through oversampling strategies. The reason why we used the Transformer-based model from the second highest ranked team and not the first is that their source code was easily accessible to us. Note, however, that the results from both teams in the shared task were very similar, with a difference of only one percentage point in the polarity classification task. Thus, we can assume that the results of this system are a reasonable representation of the state of the art.

For the comparison with the Transformer-based model, we use the following dictionaries: (1) words with low ambiguity extracted from short and long titles from the Rest-Mex training data as explained before (Kellert, Gómez-Rodríguez & Uz Zaman, 2024) and (2) a combination of dictionaries: the Spanish version of SO-CAL, together with words from the VADER dictionary that are missing in English SO-CAL that we translate into Spanish via DeepL (sp. VADERdict), and words with low ambiguity extracted from short and long titles from the Rest-Mex training data (Kellert, Gómez-Rodríguez & Uz Zaman, 2024).

One of the reasons why we experiment with several dictionaries and their combinations is to see which dictionary or combination produces better accuracy results in polarity prediction tasks, both within the same system and across different systems.

For the evaluation of the systems on the polarity prediction task, we converted the outputs of these systems into class labels for ternary classification ([positive, neutral, negative]) (see “Appendix A” on conversion rules), which we then compared with the gold polarity classes obtained from the datasets in the same format (Data). As an exception, for the comparison with the Transformer-based model, we perform the evaluation directly on the five-star scale (i.e., 5-class classification) due to the restrictions inherent to this evaluation: as mentioned before, the model has been trained and adjusted on the training set, so we must use the test set to which we do not have direct access, performing instead the evaluation by sending system outputs to the task organizers. Therefore, no conversion is possible.

Results and discussion

We first report our results for dependency parsing, which serves as the basis for the subsequent polarity classification. The parsing results in Table 4 include both overall parsing metrics (UAS, LAS, MLAS and BLEX) and fine-grained metrics for some specific dependencies that are especially important for sentiment analysis.⁴ In the overall metrics, our SELSP falls short of state-of-the-art accuracy for the UD EWT English and UD Spanish AnCora treebanks, and is also less accurate than Stanza, achieving a few points over 75% LAS. However, if we look at the detailed results, the picture is different. In spite of Stanza’s higher LAS, SELSP is highly effective on specific dependency types that are crucial for sentiment analysis. For example, it is more accurate at detecting adverbial modifiers (advmod), subjects (nsubj), and objects (obj) across both English and Spanish. These dependencies are vital to capture intensifiers (e.g., “very good”) and connect the sentiment to the subjects and objects involved (e.g., “I love this book”).

In this respect, it is worth reminding that syntax-based polarity classification does not need high parsing accuracy (Gómez-Rodríguez, Alonso-Alonso & Vilares, 2019) so our goal here is to optimize for speed (hence using a sequence-labeling approach and a small distilled language model, decisions that sacrifice accuracy in favor of efficiency). This goal is evaluated in Table 5, where we report the speed of the whole system (including dependency parsing and polarity classification). Indeed, our results show that our SELSP-based SA system is much less time-consuming than the alternative using the conventional syntactic parser Stanza as well as the VADER system, as it processes almost three times more sentences per second than Stanza and 18 times more than VADER. Speed does not vary substantially with respect to the sentiment dictionary used, with only a slight slowdown when using larger dictionaries. SELSP’s speed advantage is attributed to its sequence-labeling formulation, which contrasts with the computationally intensive ad hoc parsing algorithms employed by Stanza. By reformulating parsing as a linear-time sequence-labeling task (see “Dependency Parsing Approaches”), SELSP achieves very fast processing speeds.

Tables 6, 7, and 8 show the accuracy results for SELSP, Stanza, and VADER for different datasets and different dictionaries (see “Methodology”). Compared to the baselines, the results demonstrate that SELSP achieves significantly better performance than VADER across all language, dataset, and dictionary combinations. This improvement stems from fundamental methodological differences: while VADER relies on shallow heuristics without syntactic analysis, SELSP explicitly models syntactic relationships (e.g., negation scope, intensifier modification) through dependency parsing, enabling more precise polarity predictions, particularly in complex sentences. The accuracy gain over VADER highlights the importance of syntactic modeling for polarity shifts. When comparing SELSP to Stanza, we observe that the accuracy of the system using the SELSP parser is not only on par with the version using Stanza, but even outperforms it in most cases (nine dataset-dictionary combinations where SELSP provides better accuracy than Stanza, vs. three where Stanza is better). This is particularly interesting, given that our goal in replacing Stanza with SELSP was to optimize for efficiency, and the SELSP parser prioritizes speed over accuracy (Table 4) and in particular is less accurate than Stanza in overall terms. The fact that this does not result on any loss of accuracy on the end task is a confirmation of the previous observation by Gómez-Rodríguez, Alonso-Alonso & Vilares (2019) that syntax-based polarity classification performance no longer improves with parsing accuracy once a “good enough” parsing accuracy threshold is achieved. Clearly, the accuracies provided by SELSP in Table 4 are above that threshold for our datasets under consideration, especially when considering that accuracies are especially good on dependencies that are crucial for sentiment analysis, so our results confirm that it is beneficial to use parsing as sequence labeling in this scenario.

If we compare dictionaries, we observe that SO-CAL provides the best results for English (and augmentation with VADERdict does not help), although differences are not large. On Spanish, there are no large differences between the different dictionaries we consider, except that the sentiment dictionaries automatically generated from Rest-Mex titles provide better results on the Rest-Mex dataset, suggesting that it is beneficial to generate dictionaries from in-domain data when possible.

In terms of datasets, our sample of Rest-Mex reviews are clearly easier to classify than the OpeNER reviews, as every system provides better accuracy on Rest-Mex than on the other datasets.

Finally, we compare our models to a supervised Transformer-based model on the Rest-Mex Spanish dataset. Results are shown in Table 9. Note that, as explained in Section “Methodology”, this comparison is performed on a different split (the Rest-Mex test set) and we perform five-class classification, rather than three class, so results are not directly comparable to those in Table 8. The table shows a clear advantage of the Transformer-based model in terms of accuracy. This result is not surprising, since supervised models are known to excel when trained on a corpus with similar texts as the test data (Vilares, Gómez-Rodríguez & Alonso, 2017), as is the case here. The use case for unsupervised lexicon-based models, like the rest of the systems considered in this article, is when no in-distribution data is available to train a supervised model, a common situation in production.

To better illustrate the strengths and limitations of our approach, we now present a qualitative analysis of specific representative examples from the test data, including success and failure cases. Our parser’s performance demonstrates that it achieves its best results in compositional cases, where the combination of word meaning and syntactic structure naturally leads to the correct sentiment outcome. In such instances, the parser effectively applies syntactic rules to interpret sentiment polarity, as illustrated in example Figs. 5, 6 and 7. For example, when a sentiment-bearing word is negated, the parser accurately reverses the polarity (e.g., recognizing that “no adequate” conveys a negative sentiment in example Fig. 5). These cases highlight the strength of the system in handling syntactically transparent constructions, where sentiment can be reliably derived from grammatical composition rather than from contextual or idiomatic interpretation.

Our parser’s performance is not independent of the sentiment dictionaries it relies on. Therefore, prediction errors are not necessarily due to failures in parsing but often result from missing or incomplete lexical entries, as illustrated by example Fig. 8. However, we note that example Fig. 9 is not strictly out-of-vocabulary (OOV), since the word “sure” is in the vocabulary. The issue lies in the low positive polarity score, which is insufficient to trigger a positive classification, leading the system to output a neutral sentiment.

Another limitation is that our approach currently associates each word with a single sentiment value, without considering contextual information or ambiguity. For instance, the word taste may be interpreted as positive in “this food has taste”, but negative in “this has a perfume taste”, where it conveys an artificial or unpleasant flavor—as seen in one of our failure cases in Fig. 10. In example Fig. 9, “we will come back” also involves implicit sentiment, since the surface meaning simply describes an action, but pragmatically it implies a positive evaluation—if one returns, it is because the experience was favorable.

Furthermore, misclassifications can occur when sentiment-bearing words are written in nonstandard forms or contain spelling errors, preventing proper dictionary matching (see example Fig. 8). Overall, sentiment analysis errors are not purely syntactic in nature nor solely dependent on the parser’s ability to identify sentiment structure; they also depend on external factors such as dictionary coverage, lemma identification, and the system’s sensitivity to context.

Conclusion

In this study, we have addressed the speed bottleneck problem of sentiment analysis systems that use syntactic parsing. For this purpose, we trained a model for syntactic parsing as sequence labeling (SELSP) based on a pre-trained language model DistilBERT within the CoDeLIn formalism for sequence labeling. We then applied it to sentiment analysis on Spanish and English datasets, by using its output parse trees to compute sentence polarities from sentiment words through a sentiment analysis engine which relies on syntax-based rules.

Our main experiments on 3-class polarity classification on a range of English and Spanish datasets, and with a variety of sentiment dictionaries, show that SELSP achieves considerably faster processing time than the same syntax-based rules applied together with the conventional syntactic parser Stanza, as well as the popular heuristic approach VADER. In terms of accuracy, SELSP achieves better results than VADER across all datasets and sentiment dictionaries tested, and it even outperforms the Stanza-based alternative in most dataset-dictionary combinations. This shows that sequence-labeling parsing is especially fitting for syntax rule-based sentiment analysis, as we achieved substantial speed gains with no cost (or even with improvements) to accuracy. While this work focuses on sentiment analysis, the SELSP framework could potentially benefit other NLP tasks reliant on syntactic parsing, such as semantic role labeling or relation extraction. Investigating these applications remains an avenue for future research.

To put our results into context, we also ran a comparison with a supervised system based on variants of the Transformer-based model RoBERTa, which achieved state-of-the-art results at the Rest-Mex 2023 shared task. The supervised system achieved worse speed, but better accuracy, than our proposal in our comparison. This accuracy advantage is expected, as supervised systems trained or fine-tuned with in-distribution data (in this case, with more than 250K instances of reviews of the same type it is being evaluated with) tend to outperform unsupervised, rule-based systems as they can adapt specifically to the characteristics of the training data. However, this trend has been shown to often invert when one does not have in-domain training data, or even when the data is in-domain but from a different source or time period (see Vilares, Gómez-Rodríguez & Alonso, 2017). For this reason, unsupervised approaches like the ones we focus on here are still popular in industry applications, where it is often not possible (or very costly) to obtain large amount of data that closely resembles production data.

The improved processing times and the improved accuracy performance in 3-class categorizations (neutral, positive, and negative), together with not needing any training data and the explainability afforded by the parse trees and syntactic rules, make SELSP very attractive for practitioners of Sentiment Analysis in research and industry.

We noticed a big difference in accuracy across datasets (OpeNER vs. TripAdvisor reviews). This difference is not specific to our proposed system, but also common to all baselines, and it may be related with the length of the reviews (see “Data”). We believe that the performance of SELSP in processing short reviews could be improved in the future by using specialized large language models that were pre-trained on short messages like tweets, such as BERTweet (Nguyen, Vu & Nguyen, 2020). As our model depends on the syntactic relation between sentiment words and polarity shifters like negation, improving these rules can also lead to an improvement in accuracy.

We also tested different sentiment dictionaries, to test their influence on polarity classification results as well as their interaction with different systems. In this respect, we observed that accuracy values differ more across models (SELSP, Stanza and VADER) than across dictionaries of the same model in OpeNER. For instance, the differences between dictionaries within the SELSP model in processing English reviews are in the 3% range, whereas the differences in accuracy between models such as SELSP and VADER are in the 10% range. Similar observations can be made for the Spanish reviews from OpeNER. This suggests that the choice of a model or approach of syntactic representation is more important for the accuracy of polarity prediction tasks than the choice of a dictionary, at least for processing reviews from OpeNER. However, if we consider the longer reviews from the Rest-Mex dataset, differences between dictionaries are comparable to those between models, as shown in Table 8.

Limitations

In our experiments, we do not perform any optimization or other task-specific tweaks to try to improve the accuracy of the polarity prediction task. This is because our main focus in this article is not to improve the accuracy of the Sentiment Analysis but rather to improve the temporal restrictions of syntax-based sentiment analysis systems to leverage their advantages such as explainability and transparency in practical settings, in contrast to purely supervised approaches. We note that, even under such a setup, the accuracy of our model is good, even outperforming the Stanza-based model apart from the VADER baseline. Another limitation of our approach is that we only use only English and Spanish to train our model (Task 1) and to measure the accuracy of the polarity prediction task (Task 2). This is because our approach requires language-specific sentiment dictionaries and language-specific words to identify polarity-shifting elements like negation and intensification. We will acquire these resources for other languages in future research to be able to include more languages for testing and evaluation. While the result that parsing accuracy beyond a “good enough” threshold is enough to help parsing accuracy was originally only shown for English (Gómez-Rodríguez, Alonso-Alonso & Vilares, 2019), our results suggest a similar trend for Spanish, making us expect that it will be generalizable to other languages, although further research is needed. Finally, we acknowledge that converting sentiment polarities to a ternary scale (Appendix A) could introduce some noise, but such conversions are unavoidable when testing on diverse datasets.

Appendix A

Appendix B