Dynamic decoding and dual synthetic data for automatic correction of grammar in low-resource scenario

Confusion method

The proposed confusion method is a combination of modified versions of two approaches. The first technique named Inverted-Spellchecker was introduced by Grundkiewicz, Junczys-Dowmunt & Heafield (2019) and the Patterns+POS method was proposed by Choe et al. (2019), both approaches achieved significant improvements in GEC in English and other Latin languages. However, the constructed data was used to train the GEC model, which left side of Fig. 1 provides a graphical representation of the process of generating synthetic data, and the following subsections provide detailed information.

Dynamic decoding

The proposed SiaGEC framework uses dynamic decoding in the decoding process, which is an algorithm capable of simultaneously generating a token and computing the subsequent token probability by incorporating the lexicon and the character models. In this way, by incorporating both models, the algorithm dynamically adjusts subword segmentation during generation. The probability calculation for each sub-word is described in the following equation: (5)${\displaystyle p\left(s_n|s_{<n},x\right)\approx p\left(s_n|\pi \left(s_{<n}\right),x\right)=p\left(s_n|y_{<m},x\right)}$ where s_<n is a sequence of subwords that is converted by concatenation operator π(s_<n) into the raw character y_<m of preceding subword s_n. The equation of p(s_n|y_<m, x) is based on incorporating the character and the lexicon as follows: (6)${\displaystyle p\left(s_n|y_{<m},x\right)=g_{m}Pchar\left(s_n|y_{<m},x\right)+\left(1-g_m\right)Plex\left(s_n|y_{<m},x\right).}$

Equation (6) above states that the subword probability is obtained from the combination of character and lexicon probabilities.

In the dynamic decoding process, we model the subword boundary decision explicitly. For instance, when we create a character, we take into account the character potential on the left (previous) and the right (next) whether to continue or end the subword as done by a naive beam search (Huang et al., 2019). This aims to avoid bias towards short subword by not directly comparing complete and incomplete subwords. We have four considerations that can calculate the probability of the next character, as follows:

• Co-End (The left character continues a subword ended by the right character)

• Co-Co (Both left and right characters continue the same subword)

• End-End (Both left and right characters end the subword)

• End-Co (The character on the left concludes a subword while the one on the right continues it (starts a new subword))

We can perform the next-character probabilities and output one character at a time if we apply subword boundaries as follows p_co−end > p_co−co or p_end−end > p_end−co. This is as described in Algorithm 1  below.

 
_______________________ 
Algorithm 1 : The proposed dynamic decoding process_____________________________ 
  1:  Input: x → The source sentence 
 2:  Output: y∗ → The generated output a character sequence concluding with < 
    eos >  (end of sentence) 
  3:  Notation: C → A character vocabulary 
 4:  yco : Partial output, last char ends subword 
 5:  yend : Partial output, last char continues subword 
 6:  Yco = argmaxPend−co(Y |X),Yco = [Yco] 
  7:             Y ϵC 
 8:  Yend = argmaxPend−end(Y |X),Yend = [Yend] 
  9:             Y ϵC 
10:  while yend[−1] ⁄=< eos > do 
11:     Yco−co = argmaxPco−co(Y |yco,X) 
12:                 Y ϵC 
13:     Yend−co = argmaxPend−co(Y |yend,X) 
14:                 Y ϵC 
15:     yco = argmaxP(y) 
16:                 yϵ[yco,Yco−co],[yend,Yend−co] 
17:     Yco−end = argmaxPco−end(Y |yco,X) 
18:                 Y ϵC 
19:     Yend−end = argmaxPend−end(Y |yend,X) 
20:                 Y ϵC 
21:     yend = argmaxP(y) 
22:                 yϵ[yco,Yco−end],[yend,Yend−end] 
23:  end while 
24:  y∗ = yend 
25:  return  y∗______________________________________________    

Essentially, the decoding algorithm will be more ideal if it makes final segmentation decisions based on the characters before and after the potential subword boundaries during the decoding process. Thus, all possible segmentation in each generation step can be considered properly. The four considerations that can compute the probability of the next character can be formulated as follows: (7)${\displaystyle P_{end-end}\left(Y|y_{<m},x\right)=g_{m}Pchar\left(Y,<eos>|y_{<m},x\right)+\left(1-g_m\right)Plex\left(Y|y_{<m},x\right).}$

This is the first and simplest case, in which the generated previous character at position m − 1 concludes a subword. Then, the single character Y is the next subword probability. <eos > is the special end of the subword token. In this context, Eq. (7) describes that all possible characters Y in vocabulary can be computed and be candidate for the next character. The second case is the modification of the Eq. (7). The case is such that the character m − 1 does not end a subword but represents the final character within the subword, as shown in the subsequent equation: (8)${\displaystyle P_{co-end}\left(Y|y_{<m},x\right)=g_{m}Pchar\left(Y,<eos>|y_{<l:m-1},y_{<l},x\right)+\left(1-g_m\right)Plex\left(Y|y_{<l:m-1},y_{<l},x\right)}$

where y_<l:m−1 are the generated character in the current subword and l is the current subword starting position that concluded at m. Equations (7) and (8) still provide candidates when the subsequent character ends a subword. To obtain the probability of the next character that starts and continues a subword, we modify Eq. (7), as the following: (9)${\displaystyle P_{end-co}\left(Y|y_{<m},x\right)=g_{m}Pchar\left(Y|y_{<m},x\right)+\left(1-g_m\right)\sum _{s:s1=Y,s\ne Y}Plex\left(s|y_{<m},x\right)}$

In Eq. (9) above, the probability of the succeeding character in the first component is exclusively dependent on the character-level model, excluding token ¡eos¿. Meanwhile, the other component omits all subwords that start with the specified character y(s ≠ Y), because it is a subword that ends with the character m. As a generalization of Eqs. (7) to (8), (9) can also generalize to the next equation, namely the case where the character m continues the subword starting with the previous character. Thus, it produces the following equation; however, the dynamic decoding algorithm process is described in detail in Fig. 2. (10)${\displaystyle P_{co-co}\left(Y|y_{<m},x\right)=g_{m}Pchar\left(Y|y_{<l:m-1},y_{<l},x\right)+\left(1-g_m\right)\sum _{s:s1=Y,s\ne Y}Plex\left(s|y_{<l:m-1},y_{<l},x\right).}$

Incorporating the Confusion Method, which combines the Inverted Spellchecker and Patterns+POS approaches, we generate synthetic data that enhances the training and evaluation of our GEC model. Dynamic Decoding, an integral part of our SiaGEC framework, further refines our model’s performance by adjusting subword segmentation during generation. These methodological components collectively empower our framework to address the unique challenges posed by low-resource languages, showcasing its potential in the realm of Grammar Error Correction.

Dataset

In this work, we sourced the seed synthetic training data from CC100-Indonesian, a monolingual dataset that Facebook AI teams (Conneau et al., 2020) prepared. The data was collected from January–December 2018 using Commoncrawl snapshots in the CC-Net repository and organized into a single text document with a total data size of 36 GB. We selected the CC100 Indonesian corpus due to its open accessibility and its status as the most comprehensive monolingual Indonesian corpus available. This extensive dataset encompasses articles from a wide range of sources, covering diverse topics such as sports, health, economics, law, history, classic tales, and culinary recipes.

However, due to our hardware limitation, we utilized 711 MB from the entire CC100-Indonesian dataset. Following the data pre-processing phase, we employed the confusion method InSpelPoS as described in the Inverted Spellchecker subsection to generate a large synthetic training data totaling 3.9 GB (source and target). The comparison of data between original and synthetic data can be up to five times larger, meaning that the dataset used for training the proposed model (synthetic) is much larger than the original dataset. Finally, we divided the synthetic training data into two sets of categories consisting of a training set of 8,681,648 examples, and a valid set of 2,170,412 examples at 80% and 20%, respectively. The test set was a benchmark from the previous work consisting of 650,129 examples.

Data preprocessing

In this context, the sustained progress of neural GEC is partly credited to the efficient handling of word representations and subword segmentation (Meyer & Buys, 2023). However, word-level techniques can give rise to numerous challenges beyond vocabulary limitations, whereas character-level methods tend to be computationally demanding and lack semantic richness (Solyman et al., 2021). To address the problem of open vocabulary, we utilized byte pairing coding (BPE) (Sennrich, Haddow & Birch, 2016b) algorithm to split and represent rare or unknown tokens into subwords (32K subwords) immediately after tokenizing the dataset with SpaCy (https://spacy.io/). We set the maximum input sequence length to 200 words for both the training and testing phases. This subword segmentation ensures consistency in tokens across training and development sets, ultimately enhancing model performance. In the inference phase, we manually select all the hyperparameters for our model based on our experience with similar models and considering the computational resources available.

Model configuration

In the experiments, we trained the proposed model using the baseline sequence-to-sequence Transformer architecture (Vaswani et al., 2017). Besides adapting the decoder block with the dynamic decoding method as detailed in the methodology section, we also implemented several modifications to the baseline architecture, including adjustments to the model size, batch size, number of layers, and head attention. Initially, we adjusted the model size from 512 to 256 and modified the batch size from 2,048 tokens to 256. We then decreased the number of layers from six to four, while retaining the original number of head attentions to eight which plays a significant role in understanding the dependencies between different parts of the input sequence, thereby improving the model’s ability to generate accurate and contextually relevant outputs. In order to boost performance, we adopted the learned positional coding technique rather than the static coding method from the original article (Vaswani et al., 2017). In line with the BERT configuration (Devlin et al., 2019), we refrained from using label smoothing. We utilized the static Adam optimizer (Kingma & Ba, 2015) with a steady learning rate of 1x10³, forgoing the warm-up and cool-down strategies. During training, we employed early stopping whenever there was no enhancement in validation data after 30 epochs. However, dropout was applied with probabilities of 0.1 and 0.15 to mitigate the risk of model overfitting. In our study, we intentionally set the hyperparameters manually. While automated methods for hyperparameter selection exist, we chose manual configuration during experiments for precise control over the models. All experiments were conducted using PyTorch in Python version 3.9 and executed on two NVIDIA Titan RTX GPUs with 25 GB of RAM each, making use of NVIDIA CUDA Toolkit 10.2.

Evaluation

We applied MaxMatch scorer (Dahlmeier & Ng, 2012) as our evaluation metric to measure word-level edits within each output, subsequently providing precision, recall, and F₁ scores. F₁ scores aim to ascertain the precision, which attempts to calculate the percentage of accurate predictions to reveal the proportion of sentences corrected for grammatical errors. In this context, we calculate precision using the following equation, which indicates the percentage of errors that the system accurately corrected out of all the suggested corrections. (11)${\displaystyle P=\frac{TP}{TP+FP}}$ where TP represents true positives, indicating errors that the system has accurately corrected, whereas FP denotes false positives, indicating errors that the system has not corrected. In this metric, a higher precision score signifies that the system corrects errors very well. On the other hand, we define recall as a good error detection technique, targeting the proportion of true positives to the sum of true positives and false negatives, as in the following equation: (12)${\displaystyle R=\frac{TP}{TP+FN}}$ where FN represents false negatives, indicating the errors that the system did not detect. In this metric, a good recall score signifies that the system detects errors well. Finally, the F₁ score serves as a mean, striking a balance between precision and recall, ultimately indicating the system’s proficiency in detecting and rectifying errors, as the following equation: (13)${\displaystyle F_{1score}=\frac{2\ast P\ast R}{P+R}.}$ These metrics are able to assess various aspects of system performance, such as accurate identification and error correction of GEC systems, providing valuable perspectives on the system’s strengths and areas for improvement.

Furthermore, we utilized the BLEU-4 score (Papineni et al., 2002) to measure the similarity between system prediction results and reference data. In the BLEU score, we used N-gram precision to compute the prediction result with reference data using a maximum size of four n-grams, as in the following equation: (14)${\displaystyle BLEU-4=BP.exp\left(\sum _{n=1}^4{w}_{n}log{p}_n\right)}$ where w_n denotes weight of n-gram (set to $\frac{1}{4}$) and p_n represents precision. A higher BLEU score indicates greater similarity between the prediction results and reference data, with a score of 1 meaning the predicted and reference sentences are identical. Thus, BLEU scores tend to be high in GEC system evaluation. A brevity penalty (BP) yields a score of 1 when the prediction is overly short. Essentially, if the length of the predicted output exceeds the length of the reference data, the BP will produce a value of 1, as explained in Eq. (15). (15)${\displaystyle BP=\left\{\begin{array}{c}1ifc>r\hfill \\ ex{p}^{1-r/c}otherwise\hfill \end{array}.\right.}$

The evaluation utilized a combination of precision, recall, and F1 score metrics to assess the system’s ability to identify and correct grammatical errors. These metrics provided valuable insights into the system’s strengths and areas for improvement. Additionally, we employed the BLEU-4 score to measure the similarity between SiaGEC predictions and reference data, offering a comprehensive evaluation of the system’s overall performance. These evaluation methods together provide a well-rounded assessment of the GEC system’s capabilities.

Impact of the synthetic data

We investigated the effectiveness of InSpelPOS for generating dependable training data across various frameworks. Table 3 provides insights into the impact of synthetic data over different architectures, including bidirectional recurrent neural network (BRNN), convolutional neural network (CNN), and self-attention network Transforemer-based (Vanilla) referred to as (GEC-Trans), as same as two common synthetic data generation techniques: Inverse Spellchecker Grundkiewicz, Junczys-Dowmunt & Heafield (2019), and Pattern-POS Choe et al. (2019). BRNN with an attention mechanism designed to focus on nearby words achieved an F1 score of 49.17 and a BLEU score of 62.07. Furthermore, GEC-CNN with an attention mechanism offers the advantage of parallel training without the need for sequential operations. This CNN architecture seamlessly merged feature extraction and classification into a single task, this approach led to an improvement in the F1 score (+8.57) and BLEU score (+9.93). We also investigated the baseline GEC-Trans, a model built upon a modified version of the Transformer architecture, allowing it to process the entire input sequence simultaneously, which achieved F1 score of 68.30 and BLEU score of 74.33. On the other hand, we investigate the reliability of InSpelPOS over two state-of-the-art confusion methods Inverted-Spellchecker and Pattern-POS. Inverted-Spellchecker aims to generate confusion sets considering only spelling errors which reported 60.66 and 72.82 for F1 score and BLUE scores, respectively. However, pattern and POS-based methods generate synthetic errors utilizing a dictionary of common edits from monolingual data, and introducing noise in verbs, nouns, and prepositions. This technique reported 64.73 F1 score and 73.43 BLUE score which is better as compared to Inverted-Spellchecker and behind our proposed InSpelPOS. These results underscore the effectiveness and reliability of the semi-supervised method for constructing training data in the context of Indonesian GEC and address the challenge of limited training data availability.

Dynamic decoding

We evaluate the performance of SiaGEC using dynamic decoding and explore Word embedding, and various BPE settings, including 30k and 1,000 WordPiece units. Figure 3 shows that the incorporation of GEC-Trans with dynamic decoding and word embedding achieved F1 score of 70.93, marking an improvement of 2.63 points. Dynamic decoding enhances the model’s capacity to comprehend and rectify errors within the broader sentence context, prioritizing critical errors while mitigating over-correction. We also investigated the impact of different BPE settings to address challenges related to unknown words. Initially, we adopted a 30k WordPiece unit configuration, consistent with previous work in Musyafa et al. (2022), resulting in performance gains of 71.36 for the F1 score and 80.71 for BLEU-4. Furthermore, we observed enhancements in the GEC-Trans + dynamic decoding model by employing a 1000 WordPiece unit BPE, raising the F1 score from 71.36 to 72.61 and the BLEU score from 80.71 to 84.43. Figure 3 presents the performance of SiaGEC with dynamic decoding under the 30k and 1k BPE settings.

Multi-pass decoding

In the realm of correcting text fraught with multiple grammatical errors, using a single-pass inference approach proves to be a formidable challenge due to the intricate nature of human language. Ge, Wei & Zhou (2018) pioneered a multi-round correction method to enhance the fluency of GEC systems. This method entails iteratively refining a sentence, addressing its errors in multiple rounds, thereby progressively enhancing its fluency. In our SiaGEC model, we have integrated this approach. Initially, we trained a new version of SiaGEC from Right to Left (SiaGEC R2L) which receives an input sentence and produces an initial correction in a single round. Instead of treating this initial correction as the final output, we feed it into the (SiaGEC L2R) model to further refine the sentence in a multi-round fashion. This dynamic process results in our ultimate prediction, thereby capturing a more comprehensive correction path. In our experimental evaluation, we have established two baseline scenarios. In the first baseline, we follow the order of correction by first utilizing SiaGEC L2R and subsequently applying SiaGEC R2L. In the second baseline, we reverse the correction order, employing SiaGEC R2L before SiaGEC L2R. The results, as demonstrated in Table 4, indicate increases of 2.28 and 3.24 points in recall and 0.42 and 1.2 points in F-score for both baselines. However, it is worth noting that the precision exhibits an average decrease of 0.89 and 0.23 points in both scenarios. This decline in precision may be attributed to a potential imbalance issue between the two correction models (Liu et al., 2016).

We evaluated the performance of the SiaGEC + multipass correction model by comparing it with recent Indonesian GEC systems. In this analysis, we considered several noteworthy systems. Lin et al. (2021) proposed an Indonesian GEC model that leveraged Part-of-Speech (POS) tags and was trained on a substantial dataset comprising 10,000 sentences, which successfully corrected 10 types of grammatical errors, achieving an F1 score of 55.10. Additionally, Musyafa et al. (2022) introduced a Transformer-based GEC system tailored for Indonesian, augmented with advanced copying mechanisms, reported 71.94 F1. Table 5, shows a comparison of precision, recall, F1, and BLEU scores, and clearly demonstrates that SiaGEC surpasses the performance of existing GEC systems.

Case study

In this section, we evaluated how well the model performs using a case study with practical scenarios. The examples presented in this real-world scenario come from the Indonesian CC100 monolingual corpus. We showcased the output produced by various versions of SiaGEC, including the baseline Transformer-based model, the Transformer with dynamic decoding, and the Transformer with both dynamic and multi-pass decoding, as outlined in Table 6. Additionally, we provided the source, target, and English translation. The provided examples have several error categories as defined by Musyafa et al. (2022), such as spelling errors in 1, 2, 5, 6, and 7. The syntactic or grammatical errors are in 8, 9, and 11. The word order errors are in numbers 3, 4, and 10. The punctuation error is in number 8. At the outset, the baseline model corrected all spelling errors that are in numbers 1, 2, 5, 6, and 7, but failed to correct word choice errors, punctuation errors, and grammatical errors that are in numbers 3, 4, 8, 9, 10, and 11. Then the second baseline (Transformer + dynamic decoding) corrected all spelling errors, grammatical errors, and several word orders, which are in numbers 1, 2, 4, 5, 6, 7, 9, and 11. However, the second model failed to correct errors in punctuation and word order, numbers 3, 8, and 10. Finally, the last model (Transformer + dynamic and multi-pass decoding) is able to correct all spelling, word order, and grammar errors but fails to correct one punctuation error, namely the comma in number 8. The good performance of our proposed model showed that it is able to identify and correct errors in Indonesian text. Meanwhile, the failure of our SiaGEC model to correct punctuation errors is not a big problem in Indonesian writing. SiaGEC is still acceptable and more effective than the first baseline model in the given example. Therefore, the proposed SiaGEC model has significant improvements over models from previous work.

Discussion

The experiments have yielded promising results and insights, but it is important to acknowledge certain aspects where our findings may not fully capture the complexities of the low-resource GEC.

The experiments of this study have been focused on specific configurations, because the performance of machine learning models can be sensitive to hyperparameter choices, further exploration of hyperparameter tuning and sensitivity analysis is warranted to identify optimal settings for various scenarios and datasets. Furthermore, SiaGEC has demonstrated effectiveness in Indonesian GEC, but its generalizability to other languages may vary. Different languages exhibit unique grammatical structures, error patterns, and linguistic challenges. Extending SiaGEC to encompass a broader range of languages and language families would provide a more comprehensive understanding of the model’s applicability. Moreover, the evaluations have primarily focused on the technical performance of the models. However, the long-term impact of language correction systems should also consider user experience, user feedback, and potential ethical implications.

On the other hand, future research should focus on incorporating real-world data to assess model performance in practical contexts. This can help validate the effectiveness of SiaGEC under more diverse and complex conditions. Furthermore, extending the model’s capabilities to correct errors in multiple languages is an exciting avenue for future research. This involves accommodating variations in linguistic structures and error types across different languages. In addition, investigating the long-term impact, user experience, and ethical implications of language correction systems is crucial. This includes gathering user feedback and addressing any ethical concerns that may arise from automated correction. As well as a more in-depth analysis of specific error types and their correction can provide insights into areas where the model excels and where it requires improvement.