Semi-supervised learning and bidirectional decoding for effective grammar correction in low-resource scenarios

System overview

In this section, we introduce the proposed GEC framework in detail and formulate the hypotheses. Initially, a novel approach was proposed to construct reliable synthetic parallel training data for GEC, as shown in Fig. 1. Furthermore, we introduce a knowledge distillation with bidirectional decoding for AraGEC based on Transformer. This technique was proposed by Zhang et al. (2022) in NMT, and we have successfully integrated it into our model.

Noise method

Despite the widespread use of Arabic on the Internet, there is still a lack of freely available training data for NLP applications such as semantic analysis (Baghdadi et al., 2022), text classifications (Masri & Al-Jabi, 2023), and automatic grammar correction. The Qatar Arabic Language Bank (http://nlp.qatar.cmu.edu/qalb/) (QALB) is the only available annotated parallel data for GEC: it consists of 20,430 examples, which is not enough to train GEC neural-based systems effectively. Furthermore, building extensive parallel training data for GEC is expensive, time-consuming, and requires appropriate tools. To this end, numerous methods have been proposed such as back-translation (Kiyono et al., 2020) and misspelling confusion sets (Grundkiewicz, Junczys-Dowmunt & Heafield, 2019) to overcome the lack of training data. However, these techniques are unreliable to construct high-quality training data containing the most common grammatical errors (training patterns) and cannot control the types of errors, rate, and distribution.

While GEC has gained research attention in recent years, there has been a neglect of error identification and classification, particularly in the context of AraGEC. Previous approaches have focused on classifying grammatical errors in English, including style errors, spelling errors, semantic errors, syntax errors, missing words, extra words, and agreement errors (Naber, 2003; Wagner, Foster & van Genabith, 2007; Yuan, 2017). In the case of Arabic, Zaghouani et al. (2014) have classified seven error types, such as spelling, punctuation, incorrect word choice, morphology, syntactic, proper name errors, and dialectal usage correction, which were utilized in Arabic GEC shared tasks (Mohit et al., 2014; Rozovskaya et al., 2015). Drawing from the existing literature, our proposed method considers five main types and 14 sub-types of errors for Arabic, as outlined in Fig. 2, which includes examples and translations. These error types serve as the foundation for our approach, ensuring comprehensive coverage of the most common errors in the Arabic language, even those that were not addressed in Zaghouani et al. (2014) study.

The seed of our synthetic data was a monolingual corpus namely CC100-Arabic, created by Conneau et al. (2020) from Facebook AI. The data was collected during January-December 2018 Commoncrawl snapshots from the CC-Net repository, and the total data size was 5.4 GB organized into a single text document. The CC100 arabic corpus was selected because it is freely available and it is the most extensive monolingual Arabic corpus. In addition, it contains various topics such as education, history, economy, law, health, stories, cooking recipes, and sport. Several steps of data prepossessing were initially applied over the given corpus, such as removing the duplicate paragraphs and spaces between lines. We decided to use 25 million examples in different lengths, between 10 to 100 words. Then, data was normalized from diacritical marks, non-UTF8 encoding, links, and mentions, and we kept punctuation, numbers, and Arabic stop words.

The performance of GEC systems was improved thanks to the monolingual data, which was used during training to provide more training patterns; this depends on the size and quality of the synthetic data (Grundkiewicz, Junczys-Dowmunt & Heafield, 2019). This article proposes a semi-supervised method for generating massive synthetic data that contains the most common types of grammatical errors in Arabic. In order to cover a wide range of errors in AraGEC, two pipelines were applied; hence the type of errors was grouped into two groups: group one includes spelling errors, sentence structure, and punctuation errors; while group two includes syntax and semantic errors, as shown in Fig. 2, which include examples, translation, and the rate. We relied on the error distribution rates in the Arabic learner corpus (ALC) (Alfaifi, Atwell & Hedaya, 2014), in which punctuation and spelling have high rates, respectively. Next are the syntax errors (middle), semantic and sentence structure errors that have the lower rate (low), respectively. This makes it easy to control the rate and distribution of each type of error.

The proposed method has two key parameters: N refers to a rate of words to be processed and has an initial value between 0 and 1, we set the value of N during training to 0.1; T is the total number of words in each input sentence. Let us begin with the first pipeline, which generates errors that include Misspelling (with four sub-classes), Punctuation, and Sentence structure (with two sub-classes). For example, in the generation of spelling errors, one method aims to tokenize the input sentence and then randomly selects a word to either delete a character or add more characters. Punctuation errors are introduced by injecting specific punctuation marks from a given list or by removing existing punctuation. Sentence structure errors are induced through a method that transforms the input sentence into a part-of-speech (PoS) tagging format, followed by one of these two operations: (1) Swapping two sentence components: the subject, object, or verb, within the standard word order of VSO (verb-subject-object) in Arabic. For example, it may entail reordering the sentence from VSO to SVO (subject-verb-object) or from VSO to OVS (object-verb-subject); or (2) removing one of the sentence structures.

The second pipeline in the proposed method tackles the most complex error types: Syntax (with five sub-classes) and Semantic (with two sub-classes). To begin, each input sentence is initially converted into the PoS format. Within this pipeline, a total of seven operations (functions) are applied at each iteration $t$. These operations include but are not limited to, the following examples: (1) deletion of determiners; (2) replacement of the subject with another word from the corpus vocabulary, resulting in a verb-subject disagreement; (3) unification of the tense to the present tense format by utilizing PoS tags, disregarding future and past tenses, thus introducing tense verb errors; (4) deletion of suffixes and affixes to create morphological errors and inconsistencies within the sentence; (5) substitution of a random word in the sentence with a word from the dataset, leading to semantic errors that confuse the reader and impact the sentence context. The proposed method is named Equal Distribution of Syntactic Errors (EDSE), and its architecture is depicted in Fig. 1. Algorithm 1 provides an overview of the procedures and data flow, illustrating a simplified representation with only six functions, whereas the actual implementation encompasses a total of 16 functions. Each main error class, such as Misspelling, Punctuation, and Syntax, consists of multiple sub-classes. During each iteration $t$ consisting of three steps, a random sub-function is chosen from each main class, forming a pipeline that incorporates the selected sub-functions. However, the proposed method is not perfect and there is room for refinement, it serves as a starting point for generating different types of errors in Arabic text.

Bidirectional decoding

The proposed AraGEC framework utilizes both forward and backward decoders in its decoding structure. Since Arabic is a right-to-left (R2L) language, the decoder that moves in a forward direction employs a mask matrix in the form of an upper triangular shape, allowing it to consider the information on the right of $y_t$. This forward decoder is referred to as the R2L decoder. The backward decoder in the regular language model perceives the sequences from left to right and is named the L2R decoder. Furthermore, a lower triangular mask matrix was used in the L2R decoder. Both given decoders are utilized to detect and correct the next token, where $t$ represents the current token index ranging from 1 to T in the sequence. The decoders operate on the range of tokens from ( $t+1$ to T) or (1 to $t-1$) given the source X and the target Y, as described by the following equations.

(1) $$logP(y|\mathbf{X};\overleftarrow{\theta })=\prod _{n=1}^{N}P(y_t|y_{t+1:T},\mathbf{X};\overleftarrow{\theta }),$$

(2) $$logP(y|\mathbf{X};\overrightarrow{\theta })=\prod _{n=1}^{N}P(y_t|y_{1:t-1},\mathbf{X};\overrightarrow{\theta }).$$

The literature of the previous work in AraGEC demonstrates that the R2L performs better than the L2R decoder as described by Solyman et al. (2022); hence, in this work the backward decoder (R2L) will be the student and the forward decoder (L2R) represent the teacher. R2L decoder learns dependencies of the output sequences from right to left, whereas the L2R learns the dependencies of the output sequences from left to right, and this is the relative future information of the R2L. Thereon, the output of both decoders (R2L—L2R) which is the probability distribution of words in each position that can be represented as complementary information of two decoding sides. This makes the model force the probability distribution of $P_{R2L}$ and $P_{L2R}$ to support each other during training to generate future information, as shown in the following equation.

(3) $$P_{R2L}(y_t=w|y_{1:t-1},\mathbf{X})\sim P_{L2R}(y_t=w|y_{t+1:T},\mathbf{X})$$where $w$ is the given token from the training vocabulary, and $t$ refers the $t_{th}$ position of the output corrected sequence. However, these decoders cannot improve equally and cannot fulfill Eq. (4) if optimized separately using the standard MLE. The L2R decoder cannot learn the global coherence from R2L and this will lead to unsatisfactory corrections. To this end, a knowledge distillation method was proposed to improve both decoders during training process and the transferred information learning across R2L and L2R decoders. Furthermore, the L2R decoder will not be used during inference so as to not affect decoding speed as compared to the conventional GEC models that used the L2R model during inference.

Knowledge distillation

The main objective of the proposed knowledge Distillation method is to incorporate the learning information from the backward decoder to the forward decoder, which uses L2R decoder as a teacher that has future knowledge (hidden states) of R2L decoder. This approach utilizes the logits and the teacher’s final layer hidden states model for increased versatility and effectiveness. Furthermore, since the student and teacher models will learn during training at the same time, so we called this method Bidirectional Knowledge Distillation Grammatical Error Correction (BKDGEC), which encompasses hidden state-based distillation and logit-based as depicted in Fig. 3. In this context, we would like to clarify that the term “knowledge distillation” typically implies transferring information from a larger model to a smaller model. However, in our framework, we use the term “knowledge distillation” to describe the transfer of knowledge from the teacher to the student decoder, even though they have similar sizes.

In the realm of neural-based techniques, Logit alludes to the predictive vector that can be produced using the last layer of the decoder. This layer has the same dimension as the vocabulary size and is employed to determine the token that should be predicted in the present time step. In this work, Kullback-Leibler (KL) divergence was utilized to quantify the divergence between the logit probability distributions of the backward and forward decoders at the same position. Eqs. (4) and (5) demonstrate the implementation of this method:

(4) $$L_{logit}=\sum _{n=1}^{T}KL(P(y_t|y_{1:t-1},\mathbf{X}\overrightarrow{\theta })||P(y_t|y_{t+1:T},\mathbf{X};\overleftarrow{\theta })),$$

(5) $$\begin{array}{rl}& KL(P(y_t|y_{1:t-1},\mathbf{\text{X}}\overrightarrow{\theta })||P(y_t|y_{t+1:T},\mathbf{\text{X}};\overleftarrow{\theta }))\\ & =\sum _{w\in V}P(y_t=w|y_{1:t-1},\mathbf{X}\mathbf{;}\overrightarrow{\theta }\mathbf{)}\mathbf{)}\times \mathbf{l}\mathbf{o}\mathbf{g}\frac{\mathbf{P}\mathbf{(}{\mathbf{y}}_{\mathbf{t}}\mathbf{=}\mathbf{w}\mathbf{|}{\mathbf{y}}_{\mathbf{1}\mathbf{:}\mathbf{t}\mathbf{-}\mathbf{1}}\mathbf{,}\mathbf{X}\mathbf{;}\overrightarrow{\theta }\mathbf{)}}{\mathbf{P}\mathbf{(}{\mathbf{y}}_{\mathbf{t}}\mathbf{=}\mathbf{w}\mathbf{|}{\mathbf{y}}_{\mathbf{t}\mathbf{+}\mathbf{1}\mathbf{:}\mathbf{T}}\mathbf{,}\mathbf{\text{X}}\mathbf{;}\overleftarrow{\theta }\mathbf{)}}\mathbf{.}\end{array}$$

Here, V represents the output vocabulary, and T denotes the target length. Consequently, this led to the distillation of hidden states, which can be depicted through the following equation.

(6) $$L_{hd}=MSE(\overleftarrow{H{W}_h},\overrightarrow{H})$$where MSE is a loss function stands to mean squared error, $\overleftarrow{H}\in R^{l\times \acute{d}}$ and $\overrightarrow{H}\in R^{l\times d}$ refers to the hidden states of the both decoders R2L and L2R, respectively. Furthermore, $W_h\in R^{\stackrel{\prime }{d}\times d}$ is a linear function that adjusts the L2R hidden states to have the same dimension as the R2L hidden states, and $\stackrel{,}{d}d$ are the hidden dimension of both the decoders and have the same value. In this work, two knowledge distillation functions were utilized to encourage the backward decoder to grasp future representations. In addition, a joint training framework was constructed to optimize both the decoders iteratively, as shown in Eq. (7).

(7) $$L(\theta )=\sum -logP(\overrightarrow{y}|X,\overrightarrow{\theta })-logP(\overleftarrow{y}|X,\overleftarrow{\theta })+L_{kd}(\overrightarrow{y},\overleftarrow{y}),$$

(8) $$L_{kd}=L_{logit}+L_{hd},$$

As explained, the knowledge distillation learning process in this work is based on a student model imitating the teacher model. This might raise concerns as the student’s potential might be constrained by the teacher’s performance, resulting in limited ability to surpass the teacher (Clark et al., 2019). Consequently, the student model could rely heavily or excessively on the teacher model. To tackle this challenge in our BKDGEC framework, we applied two distillation methods. These methods help the R2L decoder gain a better understanding of future knowledge and drive the model to place more emphasis on the L2R decoder as training progresses. To this end, an annealing mechanism was proposed that is fitting for BKDGEC. It adjusts the training objective to consider the agreement between both decoders as in Eq. (9).

(9) $$L(\theta )=\sum _{i=1}^n\left[-(1-\lambda )\cdot {\left(\log P_{\overleftarrow{\theta }}(y_i|X_i)\right)}^2-\lambda \cdot \log P_{\overrightarrow{\theta }}(y_i|X_i)+(1-\lambda )\lambda \cdot L_{kd}(y_i,\widehat{y_i})\right],$$where $\lambda \in [0,1]$ is a hyperparameter that controls the balance between the forward decoder $P_{\overrightarrow{\theta }}$ and the backward decoder $P_{\overleftarrow{\theta }}$. Here, $y_i$ is the ground truth label for the $i$-th input sample $X_i$, and $\widehat{y_i}$ is the output label from the forward decoder. The value of $\lambda$ is determined based on the current training step $c_{step}$ and the warm start step $w_{step}$. Specifically, if $c_{step}<w_{step}$, then $\lambda =1$, and the training objective function only considers the output of the forward decoder $P_{\overrightarrow{\theta }}$ to help the backward decoder $P_{\overleftarrow{\theta }}$ acquire sufficient knowledge. Otherwise, $\lambda =\frac{w_{step}}{c_{step}}$, indicating that the number of training steps is greater than $w_{step}$. In this case, the effect of the backward decoder $P_{\overleftarrow{\theta }}$ (also known as the teacher) increases, and the initial value of the divergence in agreement $L_{kd}(y_i,\widehat{y_i})$ also increases during training, while the output of the forward decoder $P_{\overrightarrow{\theta }}$ (also known as the student) decreases over time.

Data

The synthetic parallel training data used in our study originated from CC100-Arabic, a dataset introduced by Conneau et al. (2020) from Facebook AI. Prior to utilization, the data underwent preprocessing, and subsequently, our proposed confusion method, EDSE, was applied to generate parallel training data, which was then divided into training and development sets. For fine-tuning purposes, we used the data from QALB-2014. This dataset provided us with 20,430 examples, making it a valuable resource for improving the performance of our AraGEC system. The QALB-2014 corpus consisted of English articles that had been translated into Arabic using machine translator, as well as the Arabic Learners Written Corpus (ALWC) introduced by Alfaifi, Atwell & Hedaya (2014). Additionally, it included users’ comments from the Al-Jazeera news platform that contained a range of grammatical errors due to the various Arabic dialects represented. A team of linguistic experts and native speakers corrected and annotated the data to ensure the accuracy and reliability of the target sentences.

Model setting

The initial baseline model used in our experiments was based on the Transformer architecture, as introduced by (Vaswani et al., 2017). However, certain modifications were made to adapt it to our specific task. The model size was reduced from 512 to 256, and the batch size was reduced from 2,048 to 128. These adjustments were necessary because our proposed model operated on chunks of 2-to-4 characters instead of whole words, which required smaller dimensions to ensure optimal performance. Additionally, the number of layers in the Transformer was reduced from six to four during our experiments, while the number of attention heads remained at eight, consistent with the original model. In the encoder and decoder, the first layers were used for positional encoding instead of the static encoding method employed in BERT (Devlin et al., 2019). However, we did not apply label smoothing in our approach. To tackle overfitting, we employed the Adam optimizer (Kingma & Ba, 2015) instead of using warm-up and cool-down steps for adjusting the learning rate. We set the learning rate to 0.003 during the training phase and reduced it to 0.001 during the fine-tuning phase. Furthermore, to prevent the gradient from exploding, we implemented gradient clipping with a threshold value of 1.0. Dropout regularization was also utilized with probabilities of 0.15 and 0.10 during training and fine-tuning, respectively. To handle the challenge of rare words, we applied the Byte Pair Encoding (BPE) algorithm proposed by Sennrich, Haddow & Birch (2016). This algorithm enables the segmentation of unknown tokens into sub-tokens, improving the model’s ability to handle out-of-vocabulary words and enhance overall performance.

Early stop was applied during training, which led to 27 epochs using the monolingual parallel synthetic data and three epochs for fine-tuning using parallel authentic data of QALB-2014. A checkpoint of the best model was created after each epoch. Due to the small chunks of input sequences, the maximum length of input sequences was set to 400 tokens in training and testing. The tokenizer was the BPE algorithm with 1,000 vocabulary size. Beam search was applied during inference with a five-beam size. The outputs of the test set have been tuned after inference using a simple data preprocessing method to remove the repetitions of words, characters, and some punctuation errors that the model failed to correct well.

In this context, we acknowledge that the hyperparameters were deliberately set manually. The decision to manually select hyperparameters was made to ensure careful consideration of the unique characteristics of the data and task at hand. We acknowledge that automated or systematic techniques for hyperparameter selection exist, but due to the constraints of our study and the need for fine-tuned control over the models, we opted for a manual approach. This allowed us to tailor the hyperparameter values to our specific context and strike a balance between model performance and generalization.

Evaluation

We evaluated the proposed framework using two benchmarks: QALB-2014 and QALB-2015, which were used for the Arabic GEC shared tasks in Mohit et al. (2014), (Rozovskaya et al., 2015). To assess the performance, we applied MaxMatch and the same tool used in the aforementioned shared tasks. This tool measured the word-level edits in the output compared to the reference sentences and provided precision, recall, and $F_1$ scores under different training scenarios. Furthermore, we applied the BLEU-4 score to evaluate the quality of the machine-corrected sentences compared to high-quality human-corrected sentences.

Impact of synthetic data

The constructed synthetic data have more diverse examples than those used to train BKDGEC. Table 1 shows the effectiveness of our EDSE method to construct more reliable data compared to previous approaches such as a semi-supervised confusion function (SCF) (Solyman et al., 2021) and a simple spelling noise method (SSNM) (Solyman et al., 2022) using the same data size for a fair comparison consisting of 250 k examples from each training set. The three synthetic sets have been used to train the baseline Transformer base without fine-tuning, BPE was applied with a vocabulary of 30k to reduce the confusion caused by unknown words during training. EDSE performed better than SCF and SSNM in the benchmark QALB-2014. This highlights the importance of multi-training patterns in the training data, in which SCF contains only spelling errors, while SSNM has more training patterns but is still limited compared to our synthetic data.

Eventually, the performance was investigated using the full synthetic data for training the model. Table 2 shows that F1 score increased +18.71 and +3.06 for QALB-2014 and QALB-2015, respectively. This emphasizes the importance and ability of producing synthetic data to raise the level and effectiveness of the GEC systems during training, and also the impact of 10k vocabulary of the BPE algorithm.

Impact of bid-knowledge distillation

The performance of different versions of the proposed GEC framework, utilizing two benchmarks, is presented in Table 2. The baseline model was a Transformer-based approach trained on the QALB-2014 authentic corpus, with slight modifications. The results demonstrate that the proposed BKDGEC regularization technique can significantly enhance the framework’s performance, as indicated by the $F_1$ scores of 69.73 and 72.08 for QALB-2014 and QALB-2015, respectively. Notably, the bid-knowledge distillation approach proved to be particularly effective in improving the framework’s performance, highlighting the backward decoder’s ability to predict the forward decoder’s concurrent states accurately. These findings have significant implications for the development of more effective GEC frameworks.

Impact of fine-tuning

BKDGEC has been carefully fine-tuned to improve its accuracy and performance. This fine-tuning process involved using the original parallel corpus of QALB-2014 and a monolingual dataset called CC-100 (https://data.statmt.org/cc-100/), consisting of 1 k clean sentences. The results of this process were presented in Table 2, which achieved the best results among all models with an F1 score of 70.29% for QALB-2014 and 73.13% for QALB-2015. The impact of fine-tuning on both datasets was remarkable, as demonstrated by the significant improvement in the model’s accuracy. Notably, the parallel corpus yielded better results, likely due to the inclusion of additional authentic examples. These findings highlight the importance of using high-quality datasets for fine-tuning language models, as it can have a significant impact on GEC performance.

Re-ranking n-best list

We applied re-ranking from NMT to enhance the performance after inference, which achieved significant improvement (Liu et al., 2016). Initially, three different models were trained on both sides (R2L and L2R) using BKDGEC method from scratch which utilized the synthetic data for training and which was tuned using QALB-2014. This enriches the hypothesis list which contains three different n-best lists with the corresponding scores of the R2L and L2R models. Each n-best list of the L2R models is passed to each R2L model to integrate both lists into a union relation resulting from the summation of the scores and reordered to obtain the k-best list, which is the final output. This notably improves the precision and F1 score, as shown in Table 2, which increases the F1 by 1.22 and 0.90 in QALB-2014 and QALB-2015, respectively. The impact of joint search in the n-best lists R2L and L2R led the system to improve the accuracy of prefixes and suffixes in the output. This approach facilitated more comprehensive exploration of potential corrections, enabling the system to capture and rectify errors specifically related to prefixes and suffixes.

Bidirectional decoding optimization

This subsection investigates the impact of the most common NMT bidirectional decoding techniques in GEC compared to bidirectional knowledge distillation.

BLEU score

In this subsection, we assess the performance of the proposed framework using the BLEU score to compare the quality of its outputs to the reference or golden sentences, as well as to the baseline performance (Transformer-based), which is an extra human evaluation. Initially, source sentences of the benchmark QALB-2015 were grouped into eight different lengths, also different settings have been used including n-grams with n from 1 to 4.

Table 4 shows that the proposed model achieved the highest scores in different lengths compared to the baseline trained using the same dataset and hyperparameters. The performance of both models gradually increased with the sentence length, and BLEU score settings changed, with our model being superior as shown in Fig. 4. Once again, this demonstrates the efficiency of the BKDGEC for low-resource GEC systems, which leads to overcome the challenge of exposure bias problem and improved performance without the need for extra resources or training additional models.

The proposed AraGEC framework has been compared with the existing approaches including MT-based, NMT-based, and hybrid systems as shown in Table 5. CLMB-1 is the best system in the first Arabic shared-task (Mohit et al., 2014) which is a hybrid system of machine-learning techniques and linguistic knowledge. SCUT is a neural-based model that employed CNN and attention mechanism. CUFE is a systematic rule-based system for Arabic text correction that achieved the best score in the second shared-task (Rozovskaya et al., 2015) of Arabic GEC. AHMADI and WATSON are neural-based models that exploit bidirectional RNN in different settings such as Fasttext pre-trained embeddings. PAJAK is a multi-lingual neural-based model tuned for GEC. In closing, BKDGEC achieves significant improvements over all AraGEC baselines in two benchmarks as shown in Fig. 5.

Case study

In this subsection, we assess the performance of different versions of the GEC framework using a real-world example. The example is sourced from the QALB-2015 test set and encompasses a total of 24 errors. Specifically, there are 18 instances of spelling errors (labeled as “sp”), five instances of syntax errors (labeled as “sy”) in error number five, and 13 instances of syntax errors (labeled as “sy”) in error number 13. Additionally, there are punctuation errors in four instances (labeled as “pt”), six instances (labeled as “pt”), 16 instances (labeled as “pt”), and four instances (labeled as “pt”). Figure 6 shows the output of the baseline, the baseline trained using EDSE data, the proposed model BKDGEC, and BKDGEC with fine-tuning. Furthermore, we provide the source, target, and English translation. Initially, the baseline that was trained using small training data of QALB-2014, corrected fifteen errors and failed to correct seven spelling and two punctuation errors.

Whereas a version of the baseline trained using our data EDSE has successfully corrected most of the reported errors except for three spelling and punctuation errors in 5 and 6 (pt) and caused a new punctuation error labeled “new”. BKDGEC model corrected all the errors except two spelling errors in 17(sp), 18(sp), and the new punctuation error. BKDGEC with fine-tuning has made significant improvements and successfully corrected all reported errors, with the exception of the punctuation error labeled as “new”.

This indicates that BKDGEC has been somewhat successful in challenging the scarcity of training data and also address the exposure bias problem. However, it still falls short of perfection as it is unable to correct certain punctuation, dialectal words, and challenging grammatical errors when the output of the test set is evaluated on a sentence-by-sentence basis. Therefore, extra effort is needed to correct the dialectal words, punctuation, and the most complex grammatical errors, taking into account the variability of punctuation usage in Arabic language due to writer’s style.