Check-worthy claim detection across topics for automated fact-checking

Check-worthy claim detection in the Arabic language

Check-worthy claim detection for the Arabic language was first tackled by ClaimRank (Jaradat et al., 2018) when training the neural network model with originally English sentences from political debates translated into Arabic. In addition, other learning models, e.g., Gradient boosting and k-nearest neighbors, were inspected for the task in Arabic of check-worthiness at CheckThat! 2018 (Atanasova et al., 2018).

The paradigm shift in the field of NLP occurred with the emergence of Transformers (Wolf et al., 2020), and bidirectional encoder representations (BERT), which became the state-of-art model for several tasks, including a range of text classification tasks (Devlin et al., 2019). This is also reflected in the works on check-worthy claim detection as most participants of the recent CheckThat Labs fine-tuned BERT models for Arabic such as mBERT, AraBERT, and BERT-Base-Arabic (Hasanain et al., 2020; Shaar et al., 2021; Nakov et al., 2022).

However, existing models focus on evaluating their performance on test sets whose topics overlap with those seen in the training data, without focusing on the challenges that different topics may pose to the task. As part of this study, we focus on the novel challenge of investigating how claim detection models perform differently for different topics.

Data augmentation for claim detection

Data augmentation techniques enable to leverage the (generally limited) existing data to generate new synthetic data through alterations. By carefully altering the original data samples, one can augment the available training data with the newly generated samples; however, data augmentation techniques may also produce noisy samples which lead to performance deterioration, and therefore creation of useful samples is crucial. Data augmentation strategies have also been successfully used to alleviate different problems, e.g., to settle an imbalanced dataset and to reduce model biases (Feng et al., 2021). Different data augmentation methods exist, which Li, Hou & Che (2022) categorised into three types: (i) paraphrasing, where the new augmented sentence has similar semantics to the original one, (ii) noising, which adds discrete or continuous noise to the sentence, and (iii) sampling, which is similar to paraphrasing but for more specific tasks.

Data augmentation techniques have been barely used in claim detection. For example, Williams, Rodrigues & Tran (2021) used a contextual word embedding augmentation strategy in their participation in the 2021 CheckThat! lab. These data augmentation techniques have however not been studied in the context of cross-topic claim detection. In our study, we propose the AraCWA model which provides the flexibility of being used with different data augmentation strategies. More specifically, we test three data augmentation strategies to evaluate their effectiveness in the cross-topic claim detection task, namely back-translation, contextual word embedding augmentation and text-generating augmentation.

Rumour detection

Related areas of research have focused on rumour detection (Derczynski et al., 2017; Zubiaga et al., 2018; Nguyen et al., 2021), which have aspects in common with claim detection as they also consist in finding sentences that need linking to evidence. Rumour detection is generally performed over breaking news events, which are of different nature compared to the claim detection task which typically focuses on other kinds of claims typically by reputable figures. Despite the commonalities between the two tasks, we are not aware of work detecting rumours across topics, not least in the Arabic language.

Problem formulation and evaluation

The check-worthy claim detection task consists in determining which of the sentences, out of a collection, constitute claims that should be fact-checked. Given a collection of sentences $S=\{s_1,s_2,...,s_n\}$, the model needs to classify each of them into one of $L=\{CW,NCW\}$, where CW = check-worthy and NCW = not check-worthy. The collection of sentences in S belongs to a number of topics $T=\{t_1,t_2,...,t_m\}$, where each sentence $s_j$ only belongs to one topic $t_i$. Where previous research has experimented with a mix of sentences pertaining to all topics spread across both training and test data, our objective here is to experiment with claim detection across topics by leaving a topic $t_i$ out for the test, using the sentences from the rest of the topics for training.

While the claim detection task has sometimes been formulated as a classification problem, here and in line with the CheckThat! shared tasks, we formulate it as a ranking task. As a ranking problem, the model needs to produce an ordered list of sentences in S, where the sentences in the top of the list are predicted as the most likely to be check-worthy claims. The task is in turn evaluated through the mean average precision (MAP) metric, which is defined as follows:

(1) $$MAP=\frac{1}{k}\sum _{i=1}^{k}A{P}_i$$

That is the average of the AP’s (average precisions) across classes, where in the case of claim detection $k=2$.

AP is in turn defined as follows:

(2) $$A{P}_c=\frac{1}{N}\sum j=1^N{P}_i$$where N is the number of instances being considered in the evaluation, whose precision values are averaged.

Ultimately, $A{P}_c$ measures the average precision on the top N items for class $c$. After calculating the AP values for both classes, they are averaged to calculate the final MAP.

Dataset

We make combined use of two claim detection datasets in the Arabic language, CT20-AR and CT21-AR, released as part of two different editions of the CheckThat! shared task. The datasets contain tweets which are labelled as check-worthy or not check-worthy. The tweets are also categorised into topics, which allows us to set up the cross-topic experiments. We show the main statistics of these two datasets in Table 2.

The CT21-AR dataset

Released as part of the CheckThat! 2021 Lab (Nakov et al., 2021), part of this dataset overlaps with CT20-AR. After removing the overlapping tweets, this dataset provides two new topics with a total of 600 tweets (347 and 253 tweets for each topic).

The combination of both datasets originally led to 17 topics (15 + 2). However, during the process of combining both datasets into one, we found that four of the topics are related to COVID-19, namely those with topic IDs “CT20-AR-03”, “CT20-AR-28 w1”, “CT20-AR-28 w2” and “CT20-AR-29”, Given the significant overlap between them, we combined these four topics into one, hence reducing the number of topics from 17 to 14. The final, combined dataset contains 8,100 tweets distributed across 14 topics. See Table 6 in Appendix A for a list and description of these topics.

Table 6:

Topics in CT20-AR and CT21-AR.

SN	TopicID	Title
1	CT20-AR-05	Protests in Lebanon
2	CT20-AR-10	Waseem Youssef
3	CT20-AR-19	Turkey’s intervention in Syria
4	CT20-AR-01	Deal of the century
5	CT20-AR-02	Houthis in Yemen
6	CT20-AR-08	Feminists
7	CT20-AR-12	Sudan and normalization
8	CT20-AR-14	Events in Libya
9	CT20-AR-23	The case of the Bidoon in Kuwait
10	CT20-AR-27	Algeria
11	COVID-19	COVID-19
12	CT20-AR-30	Boycotting countries and spreading rumors against Qatar
13	CT21-AR-01	Events in Gulf
14	CT21-AR-02	Events in USA

DOI: 10.7717/peerj-cs.1365/table-6

The CT20-AR and CT21-AR datasets differ slightly in the labels. Where the former only provides binary labels for check-worthy (CW) and not check-worthy (NCW), the latter provides two levels of labels: (i) claim (C) or not claim (NC), and (ii) check-worthy (CW) and not check-worthy (NCW). In both cases, we only rely on the CW/NCW labels for consistency.

While inspecting the datasets, we found the average length of tweets in CT20-AR and CT21-AR datasets together is 178 charactersr. The distribution of labels in the entire dataset is clearly skewed towards the NCW class amounting to 71.6% of the tweets, whereas the CW class only represents 28.4% of the tweets.

Further drilling down into the distribution of CW and NCW across the different topics, Fig. 1 shows a high degree of variation. With only one topic where the CW class is larger than the NCW class (i.e., CT20-AR-19, “Turkey’s intervention in Syria”), all of the other classes have different degrees of prevalence of the NCW class. Some of the topics have a particularly low number of CW instances, which is the case of “CT20-AR-08” (i.e., “Feminists”) with only 5% of check-worthy claims, and “CT20-AR-30” (i.e., “Boycotting countries and spreading rumors against Qatar”) with only 3% of check-worthy claims.

Experiment settings

We perform experiments in two different settings: (i) first, in zero-shot detection settings using AraCW, where the target topic is unseen during training, and (ii) then, in few-shot training settings using AraCWA, where a few samples of the target topic are seen during training.

To enable a fair comparison of experiments in both settings, we first hold out the data to be used as test sets. From each of the 14 topics in the dataset, we hold out 200 instances which will be used as part of the training data in the few-shot settings. Holding out these instances for all the experiments enables us to have identical test sets for both settings, where the 200 held-out instances are not used.

Claim detection models: AraCW and AraCWA

We next describe the two different models we propose for our experiments. We propose AraCW as the base model for the zero-shot experiments, as well as the variant AraCWA which incorporates a data augmentation strategy on top of few-shot learning to boost performance across topics.

AraCW: zero-shot detection across topics

Table 3 shows results for the zero-shot experiments using the AraCW model for the 14 topics separately. We can observe high variation in terms of MAP scores for the different topics, with values ranging from as low as 0.24 to 0.75. Where the average performance for all topics is over 0.54, we observe that five of the topics perform below this average (underlined and coloured in red in the table).

The best score is 0.7538 for the topicID “CT20-AR-19” which is about “Turkey’s intervention in Syria” while the worst one is 0.2468 for “CT20-AR-08” which includes tweets about “Feminists”. In the first case, we assumed that the best performance is because of the nature of the political war on the topic, and there are some topics of the same nature that were used through training such as “Houthis in Yemen” and “Events in Libya”. Thus, the model already learned some implications about the topic prior to testing. In the second case, the topic of “Feminists” is totally different and unique with respect to the other topics in the dataset, most of which are political. Likewise, the model performed poorly with topicIDs “CT20-AR-10” “CT20-AR-23” “CT20-AR-30” where the topics are about: “Waseem Youssef” which refer to religious views, “The case of the Bidoon in Kuwait” which relates to a social issue, and “Boycotting countries and spreading rumours against Qatar”. The other results fluctuate between 54% and 68%.

This indicates that the AraCW model struggles in the zero-shot settings, particularly when the target topic differs substantially from the ones used for training. This motivated our development of AraCWA leveraging few-shot learning and data augmentation, whose results we discuss in the next section.

Few-shot detection with data augmentation

Table 4 shows results for the AraCWA model leveraging few-shot learning and data augmentation, namely back-translation (BT), contextual word embedding (CWE), and text generation (TxtGen). We observe substantial improvements of the three AraCWA variants over the AraCW baseline model, with overall improvements ranging from 9% with AraCWA/TxtGen to 11% with AraCWA/CWE. These results demonstrate that AraCWA’s enhanced ability to incorporate few shots and augment data leads to substantial improvements.

When we look at the improvements across the individual topics (indicated as percentages within brackets reflecting the absolute improvement over AraCW), we observe that most of the differences are positive. Exceptions include the topicID “CT20-AR-05” where performance drops by 1% and the topicID “CT20-AR-23” about “Bidoon in Kuwait”, where the performance difference is negligible.

Apart from these exceptions, the rest of the topics experience a positive impact with the use of AraCWA. Most remarkably, four of the topics experience an improvement above 20%, which are coloured in blue. Looking back at the topics that proved problematic with the use of the AraCW model (highlighted in red in the AraCW column), we observe that the topicID “CT20-AR-10” (“Waseem Youssef”) improves by 3% with AraCWA/CWE, the topicID “CT20-AR-08” (“Feminists”) improves by 24% and the topicID “CT20-AR-30” (“boycotting countries and spreading rumors against Qatar”) improves by 21%. The remainder topic, “CT20-AR-23” (“Bidoon in Kuwait”) has no improvement (0%). Other topics with remarkable improvements include “CT21-AR-01” (“Events in Gulf; Saudi-Qatari reconciliation and Gulf summit”) which improves by 25%, and the topicID “CT20-AR-12” (“Sudan and normalization”) with an improvement of 28%. All in all, the overall 11% improvement of AraCWA/CWE shows the great potential of AraCWA compared to AraCW, which almost certainly guarantees that performance will not drop, but its improvement margin varies substantially.

Ablation experiments: few-shot learning without data augmentation

As part of the first ablation experiment to test the different components of AraCWA, we test it with the data augmentation component. Table 5 shows the results comparing the best version of AraCWA using the CWE data augmentation with the ablated AraCWA which does not use any data augmentation strategy.

We observe that the use of the data augmentation strategy leads to an overall improvement of 1% over the non-augmented baseline. There are three topics where the data augmentation makes a major impact, with improvements of 11% and 14% for CT20-AR-08 and CT21-AR-01 respectively, and a drop of 11% for CT20-AR-23. The rest of the topics experience a lesser impact, with changes ranging from −4% to +5%.

Ablation experiments: using fewer shots

We next test another ablated version of AraCWA, where we reduce the number of shots fed to the few-shot component from the original 200 shots. In this case, we test with 50, 100, 150 and 200 shots. Figure 2 shows the results for AraCWA with different numbers of shots. While there is some inconsistency between the models using 100 and 150 shots, we see that the model with 50 shots performs clearly the worst, whereas the model with 200 shots performs the best. These experiments show the importance of the few-shot learning component of AraCWA, which contributes the most to the improvement of AraCWA over AraCW.