Comparison of various approaches to tagging for the inflectional Slovak language

Motivation

Manual assigning a POS tag to each word in text is very time and labour-consuming. This leads to the existence of various approaches and methods to automate the task, where the overall process takes a word or a sentence as input, assigns a POS tag to the word or each word in the sentence, and creates a tagged text as the output (Hladek, Stas & Juhar, 2015).

Only few POS tagging algorithms and tools exist which can be deployed for low-resource languages and inflectional Slavic languages. It motivated us to conduct our research, focusing on the efficiency of these algorithms and tools especially for the Slovak language, which does not only belong to above-mentioned language families, but it is also one of official European Union languages.

The aim of this article is to compare six different automatic taggers for the inflectional Slovak language. We attempt to find the best performing tagger in terms of accuracy.

There are some studies focusing on evaluation of new proposed taggers. For example, Straka (2018) evaluated contextualized embeddings (UDPipe2) on 54 languages in POS tagging or Qi et al. (2020) built on the highly accurate neural network components that enable efficient training and evaluation for more than 70 languages (Stanza). Even though a few of them focus on Slovak, they do not compare the taggers mutually and do not distinguish between literary and non-literary texts. Moreover, they evaluated tagger accuracy using the F1 measure for the entire tagset, which provides only an overall accuracy score with gold tokenization, but lacks detailed linguistic information. In our research, we evaluated the accuracy for each position within a 15-positional tagset, enabling us to capture various grammatical aspects of the language. By categorizing the texts into literary and non-literary style, we were able to compare taggers across different text styles, a comparison that none of the studies undertook. Therefore, this study attempts to fill this gap in literature and research focusing on POS taggers for the Slovak literary and non-literary texts.

Contribution

The theoretical contribution of our research consists in designed methodology, how to compare automatic taggers with different output formats (POS tags).

The practical contribution lies in the verification of the effectiveness of six available automatic taggers for an inflectional Slovak language.

For automatic linguistic annotation of the Slovak text we recommend to use RNNTagger and UDPipe2 regardless of text type. Both tools best capture the inflection of the Slovak language. Moreover, our results indicate that for linguistic analysis of non-literary texts, the best approach is to combine RNNTagger with UDPipe2 (the latter mentioned mainly when determining tense and negation). However, for linguistic analysis of literary texts, the best approach is also to combine these two taggers, but in reverse order, i.e., only in the case of number and person we should prefer the RNNTagger over UDPipe2.

The structure of the article is as follows. The POS tagging algorithms section briefly describes principles of rule-based taggers and stochastic taggers. The Related Work section summarizes POS tagging for low-resource languages and Slavic languages. The POS taggers section describes the most known and used taggers, suitable for POS tagging of Slovak texts. The Materials and Methods section covers used dataset, selected automatic annotation tools, and applied research methodology. The Results and Discussion sections focus on the research results and their interpretations based on the performance of the taggers in terms of accuracy. The Conclusion section summarizes our findings.

POS tagging algorithms

Currently, the process of morphological language analysis is often performed in two steps. The first step is the analysis itself, which involves assigning to each word a list of possible combinations of lemma and morphological tags. The second step is unification, where one (if possible, correct) combination of lemma-tag is selected. The analysis typically consists of selecting entries from a database of inflected word forms, followed by guessing the lemma and/or tags for words outside the dictionary. The second step is often executed using statistical methods, which require training on manually annotated corpora (Garabík & Šimková, 2012).

Algorithms providing POS tagging can be grouped into rule-based taggers and stochastic taggers. Rule-based taggers—require lexical knowledge—involve a large database of handwritten disambiguation rules based on the formal syntax of the given language; on the other hand, stochastic taggers—demand high computational resources—resolve tagging ambiguities using a training set to calculate the probability that a given word has a particular tag in a specific context (Izzi & Ferilli, 2020). Stochastic POS taggers do not rely on syntactic analysis of the input, but on the Hidden Markov model (HMM), which captures the lexical and contextual information.

HMM is a doubly stochastic process with a basic probabilistic process that is not observable (it is hidden), but it can only be observed through another set of stochastic processes that create a sequence of observed symbols (Rabiner & Juang, 1986; Jurafsky & Martin, 2020). Information about the model’s state can be obtained from the probability distribution within possible output tokens, as each state of the model creates a different distribution. The sequence of output tokens provides an overview of the state sequence in the process known as pattern theory. However, algorithms associated with HMMs are efficient for performing tasks in many real-time systems; they are often applied in speech recognition, signal processing, and in some low-level NLP tasks such as morphological annotation, information extraction from documents, or speech-to-text conversion in speech recognition (Fink, 2008). In HMM words are treated as the observed events (e.g., words, that could be seen in the input) and hidden events (e.g., their parts of speech), which can be considered as causal factors in the probabilistic model (Blunsom, 2004). We treat POS tags as hidden events and individual words as observed events.

HMM consists of triple parameters $\lambda =\left(A,B,\pi \right)$ defined on the set of states Q and emissions V (Blunsom, 2004). Let Q be a set of states $Q={\left\{q_i\right\}}_{i=1}^N$ and V be a set of emissions $V={\left\{v_i\right\}}_{i=1}^N$ where π is the probability distribution function denoting the probability $\pi \left(q_i\right)=P\left(S_1=q_i\right)$, A is an N × N  matrix (called the state transition matrix), where entry a_ij is given by $a_{ij}=P\left(S_{t+1}=q_j|S_t=q_i\right)$ and B is an NxM  matrix (called the emission matrix), where entry $b_{ij}=P\left(O_t=v_j|S_t=q_i\right)$. The parameters within HMM can be estimated from the tagged or untagged words or tokens.

Related work

Tagging text in a highly inflectional language is a complex task due to word ambiguity, resulting in many homographs and, due to segmentation, into a set of morphemes (Alosaimy & Atwell, 2018). Morphological tagging offers basic information about the grammar, and/or text structure and relationships among words within the sentence. Low-resourced morphological tagging is gaining increasing recognition (Afanasev, 2023). The current shift to a language-independent approach for morphological disambiguation is regarded as an extension of POS tagging, jointly predicting complex morphological tags (Toleu, Tolegen & Mussabayev, 2022).

There are different pre-trained monolingual and multilingual models that are used for the morphological tagging, but most of them are too universal or underprepared for low resource languages, expect for the Stanza tagger (Afanasev, 2023). That is why we also decided to employed Stanza in our research. Afanasev (2023) compared Stanza to UDPipe taggers for Belarusian-Khislavichi and also for Russian-Taiga languages, all belonging to the East Slavic family, and found that a modified Stanza tagger provides more effective tagging than UDPipe. Ljubešić & Dobrovoljc (2019) conducted an experiment with three Slavic—Slovenian, Croatian, and Serbian—morphosyntactic taggers and compared two state-of-the-art tools with different architecture, traditional Reldi-tagger with a modified neural Stanza (stanfordnlp+lex). They showed that the neural Stanza yields significant improvements in tagging compared to the traditional approach. Fehle, Schmidt & Wolff (2021) evaluated two POS-taggers for German—TreeTagger and Stanza. Concerning POS-tagging, they showed very few differences.

Spoustová et al. (2009) focused on evaluating the quality of morphological annotation generated by several different POS taggers. The quality assessment was conducted for the tools HMM tagger, Morče (the predecessor of the MorphoDiTa tagger), and Feature-Based Tagger. The results of the POS taggers were categorized into three cases: correct annotation, incorrect annotation, and vague annotation. The main contribution of the research was the methodology for identifying problematic tags without the need for a human-annotated baseline.

Rosen et al. (2014) dealt with the annotation scheme of texts produced by non-native speakers of Czech. The authors not only focused on manual annotation but also conducted experiments with automated linguistic annotation tools. The results were compared for a spell checker Korektor (Richter, 2010) and the POS tagger Morče, aiming to identify errors in automated tagging. Furthermore, the authors compared two taggers: Morče and TnT (Brants, 2000). The results demonstrated that TnT faced challenges in a context with many errors, but performed better than Morče on faulty forms. On the other hand, Morče exhibited a strong preference for verbs and demonstrated better overall performance.

Machura et al. (2019) compared the Czech morphological taggers MorphoDiTa and Majka Tagger. The experimental results indicated higher precision and recall for MorphoDiTa. During the experiment, the authors enhanced the MorphoDiTa tool and significantly improved its accuracy. The authors examined the differences in the Czech language. However, the conclusion of the study emphasized that the input text has a significant impact on the quality of the output.

Straka & Straková (2017) compared different versions of UDPipe and its subsequent enhancements. The taggers were evaluated using the TIRA platform for the CoNLL 2017 UD Shared Task, where all inputs were plain text, and the results were based on F1 scores. Overall, the system upgrades demonstrated improvements in POS tag annotation. The authors utilized the old version of UDPipe as a baseline. Straka (2018) continued to improve the model by refining the model architecture, resulting in enhanced performance. The evaluation was not limited to POS tagging but also included lemmatization for the tool in the CoNLL 2018 UD Shared Task. The results of the competition motivated the author to further enhance the tool in the future.

POS taggers

MorphoDita

The Morphological Dictionary and Tagger (MorphoDiTa) is an open-source tool for morphological text analysis of natural language. It was developed within the LINDAT project by Straka & Straková (2014). It is one of the most widely used tools for morphological analysis focusing on English, Czech, and Slovak. It is based on a combination of rule-based algorithms and machine learning. Besides morphological analysis, it performs morphological generation, tagging, and tokenization. It is distributed as a standalone tool or a library, along with trained linguistic models (ibid). Its predecessors are the tagging library Morče and Featurama (Spoustová et al., 2009). The tagger is implemented as a supervised, averaged perceptron. It further utilizes two main machine learning algorithms:

• Morphological analyzer (Independent Feature Selection Classifier) for distinguishing various morphological features (e.g., number, case, gender, etc.). The classifier is trained on a large source of annotated data.

• Dependency parsing for deep syntactic analysis. It identifies relationships between words/tokens in the text and creates a tree structure of dependencies.

MorphoDiTa estimates regular patterns based on affixes, common morpheme endings, and automatically groups them into morphological “templates” without language-specific knowledge. MorphoDiTa Online operates on the same principle as the library itself, available for various operation systems (Linux/Windows/OS X) and various programming languages (C++, Python, Perl, Java, and C#). The trained model is available from 2017, and extensive changes and updates have only taken place within the Czech models.

POS categories

The part of speech category (POS) is fundamental, as each morphological interpretation of a word form is assigned a POS value (Petkevič et al., 2019). Tagsets for different languages are typically different. They can be entirely different for unrelated languages and very similar for related languages, but this is not always the rule. Tagsets can also vary in levels of granularity. Basic tagsets may contain tags for the most common parts of speech (N for noun, V for verb, A for adjective, etc.) (Universal Dependencies contributors, 2022). However, it is more common to go into detail and differentiate between singular and plural nouns, verb phrases, tenses, aspects, voices, and more. Petrov, Das & McDonald (2012) proposed a tagset consisting of twelve universal POS categories for 22 different languages, including Czech, but not Slovak: NOUN (nouns), VERB (verbs), ADJ (adjectives), ADV, (adverbs), PRON (pronouns), DET (determiners and articles), ADP (prepositions and postpositions), NUM (numerals), CONJ (conjunctions), PRT (particles), ‘.’ (punctuation marks) and X (abbreviations or foreign words). Hajic developed the Czech Prague Dependency Treebank with its tagset format (PDT tagset), designed for the needs of Slavic languages (Hajič, 2006; Bejček & Straňák, 2010). The PDT tagset uses three layers of annotation—morphological, syntactical, and tectogrammatical. It is a string of 15 characters that can more precisely determine the meaning of the tagged word; one character symbol encodes one morphological category (Hajič et al., 2020). The PDT tagset consists of a fixed length, and each position encodes one grammatical category, while two positions (13th and 14th) are empty. The attributes in positional tags are as follows: the 1st position—part of speech, 2nd—a detailed part of speech, 3rd—gender, 4th—number, 5th—case, 6th—possessor’s gender, 7th—possessor’s number, 8th—person, 9th—tense, 10th—degree of comparison, 11th—negation, 12th—voice, 13th—empty, 14th—empty, and the last position, 15th—variant and style.

According to general principles, two main groups of tagsets are recognized—UPOS and XPOS. The Universal POS tag (UPOS) is represented by tags indicating the basic categories of parts of speech. Universal POS tags are categorized into open class words, closed class words, and others. Under open class words, words can be continually added and modified. Over the decades, words like “smartphone”, “selfie”, or “e-sport” have been added to nouns. On the other hand, closed class words represents a group of words that will be universally valid and immutable (Universal Dependencies contributors, 2022). Under the others category, various symbols, punctuation marks, or symbols for unrecognizable words that, for example, the model cannot identify, are recognized.

The abbreviation XPOS refers to a language-specific POS tag specific to a given language (e.g., English: Language-specific POS). Unique rules for encoding XPOS are defined by each library. A single character, which can be a letter of the Latin alphabet, a digit, or a mathematical symbol, represents the values of individual categories. Consequently, each letter corresponds to only one value, even across parts of speech. An exception is observed in the paradigm category, which reuses the part of speech code (Garábik & Bobeková, 2021). One tag (label) for one token and lemma is formed by a set of these characters.

Dataset

Obtaining a dataset that both, best captures the diversity of the Slovak language and is sufficiently complex for evaluating tools specialized in the Slovak language tagging was challenging. Manually annotated data are crucial for training and evaluating statistical tools such as POS taggers and lemmatizers (Proisl et al., 2020). For this purpose, a manually annotated and lemmatized sub-corpus of the Slovak Dependency Corpus (SDC) was chosen (Gajdošová & Šimková, 2016). The dataset consists of 10,604 sentences and 106,043 tokens. The annotation follows the guidelines of the Prague Dependency Treebank (PDT) (Hajič, 2006) slightly modified to align with Slovak grammatical rules. Morphological tags, lemmas, and dependencies were manually assigned to each word. The sub-corpus includes only sentences in which two human annotators perfectly agreed on the tag. A drawback of the dataset is that it mainly contains short sentences (Benko & Benková, 2022). The dataset also only includes surface-dependent (analytical) trees and does not encompass a deep syntactic/semantic (tectogrammatical) layer (Majchráková et al., 2014). The primary starting point for the Slovak annotation is the functional-generative approach. Unlike the PDT, which exclusively contains journalistic texts, the stylistic-genre structure of SDC is more diverse (Šimková & Gajdošová, 2008). The texts were divided into two text types: non-literary and literary. Literary texts consist of novels and fairy tales. Non-literary texts consist of historical texts, texts obtained from Wikipedia, and journalistic texts.

The data file is available in a specific dictionary-like format (CoNLL-X format) used for text processing and morphological annotation. In the CoNLL-X format, each word in a sentence is represented as a row with various columns of information, including the word form, part of speech, lemma, and syntactic head. Every dictionary created for machine learning and deep learning is stored in this format.

A more detailed description of individual columns can be found in Gajdošová & Šimková (2016). For research purposes, only “POSTAG” column was investigated, while it contains manually annotated words from the “FORM” column. This type of data cannot be analyzed using the automatic tools, so it was necessary to reconstruct these data to the original format. The individual words were extracted from the file and reconstructed into sentence structures, which were then implemented as input for the examined automatic taggers.

Taggers

Open-source libraries or tools for morphological tagging were chosen as taggers for the Slovak language. However, upon closer examination, it was revealed that another library is already utilized by a given library or tool to process NLP for the Slovak language. Among such online tools, Sparv by Swedish developers (Hammarstedt et al., 2022) can be mentioned; it uses the Stanza tagging library for morphological annotation of Slovak. Their priority was primarily the analysis of the Swedish Språkbanken corpus. Another library is the GENIA Tagger by Tsuruoka et al. (2005), which annotates the Slovak language, but is only intended for biomedical texts, as is the STEPP Tagger (Piao, Tsuruoka & Ananiadou, 2009), which originated at the University of Tokyo and was further presented in 2012. Lastly, the Turku Neural Parse Pipeline by Finnish authors (Kanerva et al., 2018) is mentioned, which offers morphological annotation for over 50 languages, but the tagset for the Slovak language was not available. Finally, a library is mentioned that morphologically annotates the Slovak language, but its implementation was unsuccessful. The first such library was Dagger, which originated at the Technical University of Košice and was based on the principle of HMM, specifically the Viterbi algorithm and binary decision trees, particularly the ID3 algorithm (Hládek, Staš & Juhár, 2012). This library could not be implemented because it was no longer supported by newer versions of the Linux operating system. Another problematic tool was the RFTagger (Schmid & Laws, 2008), based on the HMM and decision trees method. It is suitable mainly for POS tagsets with many fine-grained tags, i.e., those that contain more details and distinguish between various subtypes of parts-of-speech and grammatical categories. Despite the excellent results achieved by this library on the German Tiger Treebank, it was not possible to annotate the entire text correctly, leading to the exclusion of RFTagger from further evaluation. For this reason, the following automated taggers were selected for comparison: TreeTagger, RNNTagger, Stanza, MorphoDiTa (application and online version will be separated), and UDPipe2.

Applied methodology

The applied methodology, inspired by other research (Hochreiter & Schmidhuber, 1997; Huang, Xu & Yu, 2015; Yao & Huang, 2016; Benkova et al., 2021; Munkova et al., 2021a; Munkova et al., 2021b; Kapusta et al., 2021), comprises the following steps (Fig. 1):

1. Acquisition of dataset with manual morphological annotation—a gold tokenization (Gajdošová & Šimková, 2016). This gold tokenization is positional and attributive, and the tags are of unequal length following inflectional paradigm. It contains 85,929 tokens.

2. Data preparation—sentence reconstruction. Since the acquired dataset was already tokenized, we converted the tokenized texts into the original text forms, i.e., to sentences, giving us 10,604 sentences (85,929 tokens). Afterwards, we divided texts according to the text type into literary texts (45,819 tokens) and non-literary texts (40,110 tokens).

3. Automatic POS tagging—converted texts (literary and non-literary texts) were annotated using the following tools:

   1. RNNTagger (using the model, trained on the SNC),

   2. TreeTagger (using the model Slovak parameter file, trained on the SNC),

   3. MorphoDiTa (using the model slovak-morfflex-pdt-170914 (Hajič & Hric, 2017), trained on SNC),

   4. MorphoDiTa online (using the model slovak-morfflex-pdt-170914 (Hajič & Hric, 2017)),

   5. Stanza (we used the model from Universal Dependencies v2.12 (Zeman et al., 2023), trained on SNC),

   6. UDPipe2 (we used the model slovak-snk-ud−2.12-230717, trained on SNC). The tools were run at two different personal computers because RNNTagger only requires the Linux operating system. RNNTagger was implemented on ASUS Transformer Book Flip TP300LD (Intel Core i5 4210 Haswell, RAM 12GB DDR3L, NVIDIA GeForce GT 820 2GB, 500 GB SSD, Ubuntu Desktop 22.10). Other tagging tools where implemented on Apple M1 Max (Processor with 10 cores 2.06–3.22 GHz, 32-Core GPU, 16-Core Neural Engine, 32 GB RAM, 1 TB SSD, macOS Ventura 13.2.1).

4. Tagset conversion—the automatic taggers (RNNTagger, TreeTagger, UDPipe2, and Stanza) do not employ the same PDT output tagset format, so their output tagsets need to be converted to the same tagset format. To compare the performance of the investigated taggers in terms of accuracy with the gold tokenization (reference), we unified the output tagset format. Since the tagset used for Slovak has tags of unequal length, i.e., a different number of characters (positions) for each part of speech, we decided to employ a universal, 15-positional tagset for all examined taggers.

5. Dataset creation—the outputs of the six automated taggers were joint into one data matrix and dummy variables were created for each position (1st–15th). These variables have a binary character, depending on the tag position match (agreement) with the reference (gold tokenization). Some positions did not contain any tags, so these positions were excluded from the experiment. We created two datasets depending on text type (literary and non-literary dataset).

6. Data analysis—we applied non-parametric procedures based on both, frequency (Cochran Q test) and ranks, where the degree of concordance was expressed by the Kendall coefficient of concordance.

Literary and non-literary texts

In the case of non-literary texts, based on the results of Cochran Q tests, the global null hypothesis (stating there are no statistically significant differences between individual tags within the tagset produced by examined POS taggers and a reference tags in tagset) is rejected at the 0.001 significance level for tags in the 1st, 2nd, 3rd, 4th, 5th, 8th, 9th, 10th, 11th, and 12th position in the tagset (1st: N = 45819, Q = 18024.87, df = 5, p < 0.001; 2nd: N = 38967, Q = 34381.00, df = 5, p < 0.001; 3rd: N = 31285, Q = 24833.86, df = 5, p < 0.001; 4th: N = 34125, Q = 16198.12, df = 5, p < 0.001; 5th: N = 32531, Q = 12497.83, df = 5, p < 0.001; 8th: N = 6672, Q = 19756.04, df = 5, p < 0.001; 9th: N = 6169, Q = 535.99, df = 5, p < 0.001; 10th: N = 7223, Q = 4400.26, df = 5, p < 0.001; 11th: N = 7377, Q = 1078.99, df = 5, p < 0.001; 12th: N = 7210, Q = 15876.04, df = 5, p < 0.001).

Similarly, for the literary texts, based on the results of Cochran Q tests, the global null hypotheses is rejected at the 0.001 significance level for tags in the 1st, 2nd, 3rd, 4th, 5th, 8th, 9th, 10th, 11th, and 12th position (1st: N = 40110, Q = 20352.07, df = 5, p < 0.001; 2nd: N = 28956, Q = 40532.13, df = 5, p < 0.001; 3rd: N = 24871, Q = 38023.70, df = 5, p < 0.001; 4th: N = 28952, Q = 18599.09, df = 5, p < 0.001; 5th: N = 21590, Q = 11327.92, df = 5, p < 0.001; 8th: N = 11077, Q = 36999.44, df = 5, p < 0.001; 9th: N = 9714, Q = 303.93, df = 5, p < 0.001; 10th: N = 4515, Q = 7028.828, df = 5, p < 0.001; 11th: N = 11779, Q = 4927.626, df = 5, p < 0.001; 12th: N = 11643, Q = 30379.50, df = 5, p < 0.001).

Our results indicate that there are differences in the accuracy of taggers’ performance, whether the text is literary or non-literary. Moreover, these differences are statistically significant.

Non-parametric procedures, which we applied, work with absolute differences of sums of ranks, where critical values were obtained asymptotically. Based on the multiple comparisons, we identified homogeneous groups among which the statistically significant differences (Tables 2, 3, 4, 5, 6, 7, 8, 9, 10 and 11) were proven.

The 1st tagset position—part of speech

In addition to the ten traditional parts of speech—noun (N), adjective (A), pronoun (P), verb (V), adverb (D), numeral (C), conjunction (J), preposition (R), interjection (I), and particle (T)—the 1st tagset position distinguishes also the abbreviation (B), foreign word (F), segment (S), isolated letter (Q), and punctuation (Z).

For both types of texts, MorphoDiTa Tagger, with less than 87% (86.65% for non-literary and 85.68% for literary texts), achieved the lowest performance in terms of matching with the reference (Table 2). On the other hand, the RNNTagger, with more than 99%, achieved the highest performance in terms of matching with the reference in the case of non-literary texts, and in the case of literary texts, UDPipe2 (99.26%) achieved the highest accuracy and/or performance with the reference (Table 2b).

For non-literary texts (Table 2a), six trivial single-element homogenous groups were identified among which statistically significant differences were observed (p <0.05).

For literary texts (Table 2b), four trivial single-element homogenous groups and one two-element homogenous group were identified among which statistically significant differences were observed (p < 0.05). In terms of agreement with the reference, the RNNTagger and UDPipe2 form one two-element homogenous group (p > 0.05).

The taggers’ performance for both text types is higher than 97% (except for MorphoDiTa and MorphoDiTa_online), which indicates that the part of speech identification (1st position in the PDT tagset format) is mostly accurate, even though the best performance was obtained from the RNNTagger for both text types (Table 2).

The 2nd tagset position—a detailed part of speech

The second tagset position contains values for fine-grained distinction of the major POS category (66 SUBPOS values) which serves as an indicator of applicability/non-applicability of other categories (Mikulová et al., 2020).

Ranking of taggers (Table 3) has again shown that the lowest performance in terms of matching with the reference in the 2nd position was achieved by the MorphoDiTa tagger (75.69% for non-literary texts and 66.30% for literary texts). Similar to the 1st position, the RNNTagger, with more than 99%, has achieved the highest performance in terms of matching with the reference for non-literary texts (Table 3a), and UDPipe2 (99.13%) achieved the highest performance and/or accuracy with the reference for literary texts (Table 3b).

For non-literary texts (Table 3a) five homogenous groups were identified among which statistically significant differences were observed (p < 0.05)—four single-element homogenous groups and one two-element homogenous group, which consists of UDPipe2 and TreeTagger.

For literary texts (Table 3b) four homogenous groups (two single-element and two two-element homogenous groups) were identified among which statistically significant differences were observed (p < 0.05). The TreeTagger and Stanza (with more than 97% of concordance) form a one two-element homogenous group (p > 0.05), and the RNNTagger and UDPipe2, similarly to the first position, form the second two-element homogenous group (p > 0.05) in terms of agreement with the reference.

The performance of the four taggers (Stanza, UDPipe2, TreeTagger, and RNNTagger) was above 96% which indicates that the identification of the detailed part-of-speech (the 2nd tagset position) is very accurate. Moreover, we can observe similar performance for above-mentioned taggers as for the first tagset position, which confirms the link (relationship) between the first and second position.

The 3rd tagset position—gender

The third tagset position denotes grammatical gender for both, lexical gender of nouns and agreement gender of verbs, adjectives, pronouns, and numerals (Mikulová et al., 2020).

Similar results of the taggers ranking for non-literary texts in the 3rd tagset position (Table 4a) have been obtained as for the previous positions. The lowest performance in terms of matching with the reference was achieved by the MorphoDiTa Tagger, with less than 75%, and the highest performance was achieved by the RNNTagger, with more than 98%. The second-highest performance, with more than 98%, was achieved by TreeTagger, which is a predecessor of the RNNTagger.

Five homogenous groups (four single-element homogenous groups and one two-element homogenous group) were identified (Table 4a), and statistically significant differences among them were observed in terms of agreement with the reference in the 3rd position (p < 0.05).

In the case of determining the tag in the 3rd position within the literary texts (Table 4b), MorphoDiTa Tagger (with less than 64%) statistically significantly performed the worst. Four homogenous groups were identified, but between homogenous group consisting of TreeTagger and Stanza, and homogenous group containing Stanza, RNNTagger, and UDPipe2, a statistically significant difference was not proven, only between TreeTagger and RNNTagger, and/or TreeTagger and UDPipe2.

Similarly, as for the 2nd position, the taggers TreeTagger, Stanza, RNNTagger, and UDPipe2 achieved the highest performance for both text styles (more than 95%). The results indicate that the RNNTagger is a suitable automatic tool for grammatical gender identification regardless of text type.

The 4th tagset position—number

The fourth tagset position has mostly two standard values—singular and plural, which are also applied to adjectives, pronouns, and numerals (Mikulová et al., 2020).

The ranking of taggers for both text types in the 4 h position (Table 5) has shown the lowest performance in terms of matching with the reference for the MorphoDiTa Tagger, with less than 86% (85.13% for non-literary texts and 83.60% for literary texts). The highest performance in terms of matching with the reference was achieved by the RNNTagger with more than 99%.

Four homogenous groups were identified for non-literary texts (Table 5a) and three homogenous groups were identified for non-literary texts (Table 5b). Statistically significant differences were observed in terms of agreement with the reference for all the taggers determining the 4th position (p < 0.05). The highest agreement was achieved for the homogenous group—Stanza, RNNTagger, TreeTagger, and UDPipe2—with more than 99% in terms of matching with the reference (p > 0.05).

The results (Table 5) show that for both text types, the best performance was achieved by RNNTagger (>99.5%). MorphoDiTa and MorphoDiTa_Online achieved the lowest performance, but still with more than 83% matches with reference for both text types (a little higher performance was achieved for non-literary texts).

The 5th tagset position—case

Slovak usually distinguishes among six (seven) cases: nominative, genitive, dative, accusative, (vocative), locative, and instrumental.

For non-literary texts, the results of the taggers’ ranking in the 5th position copies the previous positions, above all the first tagset position (Table 6a). The lowest performance in terms of matching with the reference was achieved by the MorphoDiTa Tagger (less than 84%). The highest performance, more than 98%, was achieved by the RNNTagger. Six trivial single-element homogenous groups were identified (Table 6a). A statistically significant differences were observed in terms of agreement with the reference for all examined taggers determining the 5th tagset position (p < 0.05).

For literary texts, we obtained different results compared to non-literary texts (Table 6b). The worst and statistically significant performance was achieved by the MorphoDiTa Tagger (less than 84%). The highest performance was achieved by the RNNTagger, which forms together with Stanza and UDPipe2 one homogenous group with more than 98% of matching with the reference (Table 6b). Overall, four homogenous groups were identified among which statistically significant differences were observed in terms of agreement with the reference (p < 0.05).

The results (Table 6) for determining the 5th tagset position show a good performance from almost all taggers. Regardless of text type, the RNNTagger proves to be one of the most suitable automatic tools for POS tagging (achieving more than 98.7%).

The 8th tagset position—person

The eighth tagset position expresses the person of verb forms (if applicable) and person of personal pronouns; and it usually takes on three standard values—1st person, 2nd person, and 3rd person (Mikulová et al., 2020).

For non-literary texts (Table 7a), taggers’ ranking determining the 8th position has shown two homogenous groups (p < 0.05)—one containing MorphoDiTa and MorphoDiTa_Online, and the second homogenous group consisting of the remaining taggers—Stanza, RNNTagger, TreeTagger, and UDPipe2. The first taggers’ homogenous group achieved the lowest performance in terms of matching with the reference (less than 40%). On the other hand, the second taggers’ homogenous group achieved a performance of more than 99% in terms of matching with the reference.

For literary texts (Table 7b), the worst and statistically significant performance was again achieved by the MorphoDiTa and MorphoDiTa_Online Tagger (less than 32%) forming one homogenous group (p > 0.05). Similarly, to non-literary texts, the highest performance was achieved by UDPipe2, Stanza, and RNNTagger (more than 99%) which form one homogenous group (p > 0.05).

The results (Table 7) indicate that MorphoDiTa and MorphoDiTa_Online lag behind in determining the person of verb forms or in person of personal pronouns compared to other taggers (less than 40% for non-literary texts and less than 32% for literary texts).

The 9th tagset position—tense

The ninth tagset position represents only verb forms, in the purely morphological sense—future, present, and past (Mikulová et al., 2020).

Taggers’ ranking determining the 9th position for non-literary texts (Table 8a) and also for literary texts (Table 8b) has shown two homogenous groups, and a statistically significant difference was observed in terms of agreement with the reference in the 9th position (p < 0.05). There was a small difference in performance between the two identified homogenous groups. The lowest performance in terms of matching with the reference was achieved by the MorphoDiTa and MorphoDiTa_online Tagger (about 98% for non-literary texts and about 99% for literary texts). The highest performance was obtained by the second homogenous group consisting of TreeTagger, RNNTagger, UDPipe2, and Stanza (more than 99.8%).

When determining tense, but in the purely morphological sense (9th position), Stanza proves to be the most effective tool with respect to the reference and regardless of text type.

The 10th tagset position—degree of comparison

The tenth tagset position is used for adjective and adverbs—positive, comparative, and superlative, apart from possessive adjectives.

Taggers’ ranking determining the 10th position for non-literary texts has shown five homogenous groups—four single-element and one two-element homogenous groups (Table 9a), and the statistically significant differences were observed among them in terms of agreement with the reference in the 10th position (p < 0.05). The lowest performance in terms of agreement with the reference was achieved by the MorphoDiTa Tagger (less than 81%). The highest performance was identified by the RNNTagger (more than 98%). Between UDPipe2 and TreeTagger was not identified a statistically significant difference in terms of agreement with the reference (p > 0.05).

For literary texts taggers’ ranking determining the 10th position (Table 9b) has shown statistically significant differences among four homogenous groups (p < 0.05). A homogenous group formed by Stanza, TreeTagger, and RNNTagger achieved a high performance of more than 97%, but a statistically significant difference among them was not observed in terms of agreement with the reference (p > 0.05). The worst performance was achieved for the MorphoDiTa Tagger (less than 63%) followed by MorphoDiTa_online (less than 65%). Compared to the non-literary texts, in which the highest performance was achieved by the UDPipe2 Tagger with more than 99%.

When determining the tenth position (Table 9), the combination of UDPipe2 and RNNTagger is shown to be the most effective way of automatically determining the degree of comparison with respect to the reference, regardless of the type of text.

The 11th tagset position—negation

The eleventh tagset position is fully inflectional category, as the negation in Slovak is expressed by a prefix—affirmative or negated. Negation belongs to verbs, adverbs, adjectives, and nouns (Mikulová et al., 2020).

Ranking of taggers for non-literary and also literary texts, determining the 11th position (Table 10), has shown a statistically significant difference in terms of agreement with the reference in the 11th position among three homogenous groups (p < 0.05). First two homogenous groups are single-element, and the third homogenous groups is formed by four taggers—TreeTagger, RNNTagger, Stanza, and UDPipe2 (Table 10). The lowest performance in terms of matching with the reference was achieved by the MorphoDiTa Tagger (less than 95% for non-literary texts and less than 91% for literary texts). The highest performance was achieved by UDPipe2 (more than 99%), but there is no statistically significant difference between UDPipe2 and TreeTagger/RNNTagger/ Stanza (p > 0.05).

When determining the eleventh position, UDPipe2 appears to be the most powerful tool, but there is no statistical difference between it and the other automatic tools (RNNTagger, TreeTagger, and Stanza) in determining negation with respect to the reference, regardless of text type.

The 12th tagset position—voice

The twelfth tagset position is only used for verb forms, mainly for verb participles (Mikulová et al., 2020).

A similar situation, as obtained in determining tense or negation, can also be observed in the case of voice.

Ranking of taggers determining the 12th position for non-literary and literary texts (Table 11) has identified two homogenous groups between which a statistically significant difference was observed in terms of agreement with the reference in the 12th position (p <0.05). The lowest performance was again achieved by the MorphoDiTa and MorphoDiTa_online Tagger (less than 54% for non-literary texts and less than 47% for literary texts). The highest performance was by UDPipe2, Stanza, TreeTagger, and RNNTagger (more than 98% for non-literary texts and more than 99% for literary texts), which form one homogenous group (p > 0.05).

The only difference which can be observed is the order of taggers (Table 11), i.e., in the case of literary texts, the most accurate tagger with a reference is UDPipe2, and in the case of non-literary texts, it is the RNNTagger.