VERA-ARAB: unveiling the Arabic tweets credibility by constructing balanced news dataset for veracity analysis

Textual English datasets

Existing datasets in the domain of fake news detection in English can be broadly classified into two main categories based on their content and features: textual datasets and social network datasets. Textual datasets primarily focus on news articles and textual content, emphasizing the analysis of linguistic and semantic information. These datasets typically consist of labeled samples of news articles, where each sample is annotated as either genuine or fake. The LIAR dataset (Wang, 2017) and the Reddit Comments dataset (Setty & Rekve, 2020) are prominent examples of such textual datasets. These datasets enable researchers to analyze the linguistic and semantic features of fake news and develop text-based classification models. Tables 1 and 2 outline the properties of the Liar and Reddit Comments datasets, respectively.

Social English datasets

On the contrary, social network datasets encompass not only textual content but also incorporate user and network features. These datasets offer a comprehensive perspective on the intricate dynamics of information dissemination and interactions within social media platforms. In addition to textual content, social network datasets provide supplementary contextual information, such as user profiles, follower networks, and engagement metrics. The inclusion of these features allows researchers to explore the influence of network structure, user behavior, and user credibility on the propagation and detection of fake news. Prominent examples of social network datasets extensively used for investigating the dissemination and veracity of rumors on social media include FakeNewsNet (Shu et al., 2020) and MediaEval (Boididou et al., 2018) datasets. These datasets have significantly contributed to the advancement of fake news detection techniques and have served as benchmarks for assessing the performance of various models. Tables 3 and 4 present the properties and characteristics of the FakeNewsNet and MediaEval datasets, respectively.

Textual Arabic datasets

The availability of Arabic datasets in the textual (articles) format for fake news detection has been relatively limited compared to the English language resources. However, efforts have been made to develop Arabic textual datasets specifically tailored for veracity analysis. One notable example is called the AFND dataset (Khalil et al., 2022), which consists of news articles collected from various Arabic news sources. Table 5 outlines the properties of the AFND dataset. Another significant dataset is AraFacts (Ali et al., 2021) the first comprehensive Arabic dataset of naturally occurring claims collected from multiple Arabic fact-checking websites, including Fatabyyano and Misbar. AraFacts consists of 6,223 claims spanning from 2016 to 2020, with each claim accompanied by factual labels and additional metadata such as fact-checking article content, topical categories, and links to posts or web pages associated with the claim (Table 6).

Existing datasets for fake news detection, while valuable, often exhibit notable limitations. Many datasets are either imbalanced, with disproportionate numbers of fake and real news samples, or limited in scope, focusing predominantly on specific content types such as news articles rather than social media posts. Additionally, datasets in Arabic are relatively scarce and frequently lack the comprehensive coverage needed for effective detection across diverse social media platforms. These gaps underscore the pressing need for a robust, balanced dataset tailored specifically for Arabic social media. The VERA-ARAB dataset addresses these limitations by offering a well-balanced and extensive resource designed to enhance fake news detection within the Arabic-speaking social media context. Its comprehensive nature and focus on social media posts not only bridge existing gaps but also provide a valuable tool for improving the accuracy and applicability of machine learning models in this domain.

Fact-checking

For the fact-checking process, interaction was established with Misbar (https://www.misbar.com/twitter), a platform dedicated to verifying Arab news and tweets using evidence-based methods. Misbar employs a rigorous fact-checking methodology, which involves a thorough investigation of claims by analyzing credible sources, consulting experts, and conducting in-depth research. Over the course of approximately 1 year, from August 8, 2022, to September 3, 2023, a total of 884 claims were extracted from Misbar’s fact-checking platform. These claims encompass a wide range of domains including sports, politics, armed conflicts, public security, and others that provide a valuable corpus for our research on fake news detection in the Arabic language over social media.

Annotation process

The annotation process involved a systematic approach to curating a balanced dataset of fake and real news tweets. To begin for each claim, relevant search terms and criteria were extracted to identify both fake and real news within a specific time range. These search terms were carefully chosen to capture a diverse range of fake and factual content. These search terms were then utilized to collect a preliminary set of tweets that potentially contained fake news or real news. To ensure the accuracy and reliability of the dataset, a manual annotation process was conducted. The initial collection was expanded by gathering additional tweets that matched the identified fake claims, as well as expanding the set of real news tweets. This approach allowed us to create a comprehensive dataset that represented a balanced distribution of fake and real news.

Throughout the annotation process, each tweet was meticulously reviewed by a team of trained annotators. The annotators carefully assessed the content of each tweet, considering factors such as the accuracy of the information presented, and the presence of any misleading or deceptive elements. Through this rigorous annotation process as shown in Fig. 2, a dataset was successfully compiled, consisting of 11,076 tweets containing fake news and 9,008 tweets containing true news, ensuring a balanced representation of both categories.

Data collection

The acquisition of actual tweets for our study was accomplished through the utilization of the X API protocol (X, 2024). This API provides essential tools and functionalities for accessing and extracting tweets directly from the Twitter platform. By leveraging the X API, a diverse array of valuable data was retrieved, including the textual content of tweets, multimedia attachments (such as images and video URLs), as well as user information and associated engagement metrics. The API’s flexibility allowed us to specify precise criteria and parameters to filter tweets relevant to our research objectives. The API was configured to extract tweets based on specific tweet IDs, effectively targeting the data collection process. The robust scalability of the X API facilitated the efficient gathering of a substantial volume of tweets for comprehensive analysis.

In addition to the textual content, the API provided access to extensive user information and metrics, including the number of followers, account creation date, and engagement metrics such as retweets and likes. These user-centric metrics allowed us to gain a deeper understanding of each tweet’s reach, impact, and potential for dissemination. By incorporating these diverse data points, an enriched dataset was compiled that not only included the content of the tweets but also detailed the characteristics and behaviors of the users who generated them. This comprehensive approach enabled a more nuanced analysis of fake news on social media, considering both the content and its context within the user network.

Data records and fields

Utilizing the X API described in the previous subsection, annotated tweets were extracted by their IDs. The retrieval API endpoint from the X platform allows for fetching data in batches, with each request returning a maximum of 100 tweets. The retrieved data is typically provided in the form of a JSON file, which adopts a dictionary structure comprising various key-value pairs. This dictionary includes distinct sections such as “data”, “includes” (encompassing “media”, “users”, and “tweets”), and “errors”.

All retrieved tweet data were stored in files and used to construct the VERA-ARAB dataset, mapping each tweet to its corresponding label by its ID. Table 7 lists all tweet-related fields retrieved from the X platform, while Table 8 details the user-related fields, providing descriptions for each field. By systematically mapping each tweet to its corresponding label, the integrity and accuracy of the VERA-ARAB dataset were ensured. This structured approach enabled a thorough and detailed analysis, facilitating deeper insights into the characteristics and dissemination patterns of fake news on social media.

Dataset statistics

Table 9 offers comprehensive statistics and summarizes the key metrics related to the dataset. It provide an overview of the collected claims, corresponding tweets, and users. It encompasses the total number of claims and their associated tweets and users.

To visualize the distribution of annotated tweets, Fig. 3A presents an overall distribution, highlighting the relative frequencies of real and fake news tweets. Figure 3B further breaks down the distribution by illustrating the annotated tweet distribution for each claim individually. These visual representations offer insights into the prevalence and distribution patterns of real and fake news within the dataset.

Additionally, Fig. 4 provides a graphical representation of the number of tweets posted by each user. Complementing this analysis, Table 10 categorizes the users based on the number of tweets they have posted in the dataset, providing further granularity in the characterization of user behavior. The table shows the analysis of user participation in the VERA-ARAB dataset reveals intriguing insights into the frequency of tweets posted by individual users. The data, represented by rows “Tweets” and “Users” provides a breakdown of the number of users who have participated in specific tweet ranges. For instance, the entry ‘>1’ indicates that 2,653 users have published more than one tweet, while ‘>2’ signifies that 1,192 users have contributed more than two tweets.

Examining the distribution pattern further, the number of users gradually decreases as the tweet count increases. This trend is evident as the tweet count thresholds increase to ‘>4’ (430 users), ‘>6’ (226 users), and ‘>8’ (133 users), respectively. As the tweet count becomes more substantial, the number of users gradually decreases, indicating a smaller subset of highly active participants. For instance, ‘>10’ captures 85 users, ‘>15’ includes 37 users, and ‘>20’ narrows down to 22 users. This analysis provides valuable insights into user engagement and behavior within our dataset. The majority of users exhibit lower levels of participation, contributing only a few tweets. However, a smaller subset of users demonstrates high levels of activity, consistently posting a significant number of tweets. Understanding this distribution of user engagement can provide further context for subsequent analyses, such as identifying influential users or detecting patterns in the dissemination of true and fake news within the dataset.

Spatiotemporal analysis

Spatiotemporal analysis refers to the examination and interpretation of data that is both spatially and temporally referenced. It involves analyzing patterns, trends, and relationships in data that vary over space and time. This type of analysis combines elements of traditional spatial analysis, which focuses on the geographic distribution of data, with the temporal dimension, which considers the evolution of data over time. In the context of the VERA-ARAB dataset, temporal patterns and geographic distributions of claims, tweets, and user locations were examined.

To perform a comprehensive spatiotemporal analysis, the available data on the timeline of the collected claims from August 8, 2022, to June 18, 2023 were utilized. Additionally, data on the timeline of tweets were collected, showing the distribution count over the years. Chart in Fig. 5 illustrates that the majority of tweets in the years 2022 and 2023 fall within the same timeline, which is expected due to the selective claims being made. Additionally, the chart reveals the existence of tweets predating the year 2022, where claims have been evaluated for their falsification. Through analysis of Twitter platform, previous tweets with similar evaluations or those confirming the validity of the claims were discovered and subsequently included in the dataset. Consequently, we have included them in the dataset.

The heat map in Fig. 6 reveals the number of users whose locations were tracked based on the data recorded in their profiles. It becomes evident that the majority of users are from Arab countries, particularly Saudi Arabia, Egypt, Kuwait, Yemen, and the United Arab Emirates. This outcome is expected due to the dataset being specific to the Arabic language. Following the Arab countries, the United States and the United Kingdom come in the next positions.

User profile analysis

Analyzing user profiles is of paramount importance in the context of the fake news detection problem. User profiles provide valuable insights into the credibility, reliability, and potential biases of individuals sharing information on social media platforms. Examining user profiles uncovers important indicators that help distinguish between user accounts and those associated with spreading misinformation.

The X platform provides public metrics about users such as followers, following, listed, and tweet count for each profile. Table 11 shows the statistics of such metrics for all user profiles existed in our dataset as well as the number of verified users along with the number of fake tweets and true tweets. Figure 7 gives an overview about the user accounts creation dates. Our database contains 13,421 users, with the majority of these accounts being created since 2009, as illustrated in the figure. It is also evident that there is variation in the number of accounts created during those years, indicating a significant diversity in the ages of those accounts.

Furthermore, the time difference in months between the account creation date and the tweet posting date was calculated for all available tweets. As shown in Fig. 8, the number of tweets was counted based on the age of the account at the time of posting. The results indicate an inverse relationship between the age of the account and the number of posted tweets. The graph reveals that newer accounts tend to have a higher number of tweets at the time of posting with the number of tweets decreasing as the account ages. This inverse relationship can be attributed to several factors, one possible explanation being that newer users on the platform are often more enthusiastic, motivated, and eager to actively participate in the social network community. As they join the platform, the novelty and excitement drive them to share thoughts, engage with others, and contribute to conversations more frequently through tweets.

Additionally, new users often have a smaller network of followers initially, which can result in a higher frequency of tweeting as they strive to establish their presence and gain visibility. They may actively seek interactions, engage with other users, and share content to attract attention and expand their follower base. This increased level of activity and eagerness to connect with others can result in a higher number of tweets during the early stages of their interactive journey. As time progresses and users become more familiar with the platform, they may settle into a regular tweeting pattern or find a balance between tweeting and consuming content. The initial excitement and motivation may gradually diminish, leading to a decrease in the frequency of tweets. Users may have other commitments or priorities that limit the time and energy they can allocate to tweeting, which can contribute to the decline in the number of posted tweets over time. It is worth noting that this observed trend is not universal and may vary among different users. Factors such as personal preferences, interests, and individual tweeting habits can influence the pattern of tweet frequency and account age. Nonetheless, the general trend suggests that newer accounts tend to exhibit higher tweet activity, while older accounts may experience a gradual reduction in the number of tweets posted.

Content analysis

Textual content plays a crucial role in extracting meaningful insights from datasets, such as the VERA-ARAB dataset, which specifically focuses on Arabic news content. When extracting tweet data from X platform, the Application Programming Interface (API) provides additional information about the content, such as annotation type and annotation text. The annotation type refers to the classification or categorization assigned to the tweet content based on specific criteria. The classification of the annotation types are organizations, people, places, and products. Figure 9A shows the count of words that annotated in each category. The annotations extracted by X platform produced 3,137 organization, 7,144 person, 11,499 place, and 221 products in the entire dataset. Figure 9B shows the word cloud figure for most frequent annotated words in the entire dataset.

By inspecting all claims and tweets in VERA-ARAB dataset, all tweets were manually annotated and classified into seven domains; religion, natural disaster, public security, armed conflict, public news, politics, and sports news as shown with their proportion in Fig. 10. A significant portion of news domain falls within sports and politics while news related to natural disasters or religions constitutes a smaller proportion in the VERA-ARAB dataset. Additionally, the balance of fake/true label annotations was investigated within each news domain. As shown in Fig. 11, all news domains are nearly balanced with respect to fake/true labels.

Sports news

Sports news is considered a significant genre of news both in the Arab region and globally. The sports sector itself has become a thriving industry with a distinct economy, particularly in recent decades. As seen in Fig. 10, this domain occupies the largest proportion of the dataset, accounting for approximately 25% with a total of 4,962 tweets (2,586 of which are fake news tweets). This can largely be attributed to the significant attention sports news garners from audiences in the Arab region. Moreover, the data collection period coincided with a notable surge in sporting events in the Arab sports arena. During this period, the first Football World Cup in the Middle East was held in Qatar in 2022. Following the World Cup, there were numerous news reports, including the high-profile transfer of the world-class football player Cristiano Ronaldo to Al-Nassr Club in the Kingdom of Saudi Arabia, among other sports-related news. Figure 12A shows the word cloud of the frequent words in sports news tweets.

Dataset limitations and biases

Despite the rigorous construction and comprehensive statistical analysis underpinning the VERA-ARAB dataset, several limitations and potential biases must be acknowledged. One significant concern is the geographical distribution of tweets within the dataset, which may not fully capture the diversity of regions across Arabic-speaking countries. As depicted in Fig. 6, the dataset shows a notable concentration of users from Saudi Arabia and Egypt. This regional imbalance could lead to potential biases, both geographically and dialectally, affecting the generalizability of the findings across different areas and linguistic contexts. Additionally, the dataset exhibits uneven representation of various Arabic dialects, with some dialects being overrepresented while others are underrepresented. This discrepancy introduces dialectal bias, which may influence the dataset’s effectiveness in accurately detecting fake news across the full range of Arabic dialects. The underrepresentation of certain dialects could limit the dataset’s ability to generalize findings to all Arabic-speaking populations. These factors underscore the necessity for cautious interpretation of the results and suggest that future research should focus on addressing these biases. Efforts to include a more balanced geographical and dialectal representation would enhance the dataset’s inclusivity and improve its utility for comprehensive fake news detection across diverse Arabic-speaking communities.

Preliminaries

This subsection provides a detailed overview of the approach. The problem statement is formally defined, and the key objectives of fake news classification model are outlined. The architectural design and core technical components of the proposed model are then described, with emphasis on the rationale behind the modeling choices. Finally, the evaluation metrics used to assess the model’s performance are discussed.

Model overview

For the fake news classification task, the VERA-ARAB dataset, which contains a balanced distribution of true and fake news labels is utilized to evaluate model performance on an equitable basis. The process begins with the extraction of appropriate comprehensive features as shown in Fig. 13. The dataset is then split into training and testing partitions, with 70% of the samples allocated for model development and the remaining 30% reserved for final evaluation. This partitioning strategy ensures that classifiers are assessed on unseen data, providing a robust estimate of their generalization capabilities. As a baseline, several established machine learning algorithms, including logistic regression and support vector machines, are applied.

To assess the performance of each classifier, a range of appropriate evaluation metrics is employed. Given the class balance in the dataset, accuracy, precision, recall, and F1-score are calculated to provide a nuanced view of the models’ ability to correctly identify both true and fake news instances. These complementary performance measures allow for a thorough examination of the strengths and weaknesses of the candidate models. A detailed explanition of each component presented in the following subsections.

Features extraction

A fundamental aspect of the machine learning pipeline involves transforming raw data into a set of attributes that can be effectively utilized by learning algorithms. Feature extraction significantly impacts the performance of learning algorithms by ensuring the quality and relevance of the input data. High-quality features capture the essential patterns and characteristics of the raw data, enabling the algorithm to learn more effectively and make accurate predictions. Conversely, poor feature extraction may result in the inclusion of irrelevant or noisy data, which can confuse the model and degrade its performance. Additionally, feature extraction often involves reducing the dimensionality of the data, helping to managing computational complexity and prevents overfitting. The features of $d$ dimensions with the following general matrix:

(7) $$X=\left[\begin{array}{cccc}{x}_1^{(1)}& x_2^{(1)}& \cdots & x_d^{(1)}\\ x_1^{(2)}& x_2^{(2)}& \cdots & x_d^{(2)}\\ ⋮& ⋮& \cdots & ⋮\\ x_1^{(M)}& x_2^{(M)}& \cdots & x_d^{(M)}\end{array}\right],d:\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}\mathrm{s},M:\mathrm{i}\mathrm{n}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{s}$$

Social features extraction

In order to conduct a thorough classification model, it is essential to first extract a comprehensive set of informative features from the raw dataset. This process of feature engineering is a critical step in transforming the unstructured social media data into a format suitable for subsequent modeling and hypothesis testing. The process begins with the extraction of a series of preliminary features directly observed in the dataset, such as fields listed previously in the “Data Records and Fields” section. Figure 14 shows the correlation matrix of those basic features.

Additional features are then derived through the application of domain knowledge and computational techniques, generating secondary attributes that may capture more nuanced aspects of user behavior and content characteristics. These latent features can be extracted from the content, such as the number of words in a tweet, number of charecters, number of hashtags, and other textual attributes. Furthermore, features can be derived from user information such as the age of the user account and the length user name. Figure 15 shows the correlation matrix of those latent features.

Prior to utilizing the extracted feature set for downstream analyses, a series of data preprocessing steps is applied to ensure the features are on a comparable scale and normalized appropriately. First, each numeric feature is standardized by subtracting the mean and dividing by the standard deviation. This z-score transformation centers the feature distribution at zero with a unit variance, mitigating the influence of features with larger numerical ranges. Additionally, for features representing inherently bounded quantities, such as sentiment scores or network centrality measures, min-max normalization is applied. This approach linearly scales the feature values to the range of [0, 1], preserving the relative ordering of observations while restricting the values within a common numeric domain. These standardization and normalization techniques are crucial for enabling fair comparisons across the diverse feature set and enhancing the numerical stability of any subsequent modeling procedures.

Classifiers algorithms

For experimentation, a selection of well-established machine learning algorithms were utilized as baseline models for comparison. The choice of these algorithms was guided by (Wu et al., 2008), which identifies and discusses the ten leading algorithms in the field of data mining.

Evaluation metrics

Evaluation metrics are essential in machine learning classification models as they provide quantitative measures to assess the performance and effectiveness of the models. These metrics help in determining how well the model is making predictions and identifying areas for improvement, ensuring that the model is robust and reliable.

The confusion matrix is a table (as shown in Table 12) that evaluates the performance of the binary classification model and contains four metrics: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). True positives are instances where the model correctly predicts the positive class, indicating successful identification of positive cases. True negatives are instances where the model correctly predicts the negative class, reflecting the model’s ability to accurately identify non-positive cases. False positives occur when the model incorrectly predicts the positive class for a negative instance. False negatives happen when the model incorrectly predicts the negative class for a positive instance.

From the construction of the confusion matrix, we utilize the following metrics. Accuracy measures the proportion of correct predictions (both true positives and true negatives) out of the total number of cases.

(21) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=(TP+TN)/(TP+TN+FP+FN)$$

Recall, or sensitivity, measures the proportion of true positive cases that are correctly identified by the model.

(22) $$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=(TP)/(TP+FN)$$

Precision measures the proportion of true positive predictions out of all positive predictions made by the model.

(23) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=(TP)/(TP+FP)$$

The F1 score is the harmonic mean of precision and recall, providing a single metric that balances both the precision and recall of the model.

(24) $$\mathrm{F}1=(2\ast \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\ast \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})/(\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})$$

Experiment (1): social features

In the first experiment, the tweets from VERA-ARAB dataset were preprocessed to extract explicit social criteria features such as retweet count, like count, quote count (Fig. 14). These features were selected for their potential to provide insights into the social dynamics and engagement levels associated with the tweets.

Several classifiers were employed to evaluate performance on the extracted features, including NBC, LR, SVM, AdaBoost, RF, GBoost, KNN, and XGBoost. The configurations for these classifiers were meticulously selected to optimize their performance. Logistic Regression was configured with a maximum of 200 iterations to ensure convergence. The SVM was set with a linear kernel accommodate the assumption of the linear separability. KNN was configured with five neighbors to balance bias-variance trade-off. For ensemble methods, the number of estimators was set as follows: 50 for AdaBoost, 100 for RF, and 100 for GBoost. These configurations were designed to leverage the strengths of each classifier, from simple probabilistic models to complex ensemble techniques, in order to provide a comprehensive analysis of the dataset’s classification performance.

In the second phase of the experiment, latent features derived from the tweets dataset were introduced. These latent features were expected to capture the nuanced linguistic patterns and contexts that explicit social criteria might overlook, such as word count, unique word count, hashtag count, url count, and others (Fig. 15). Finally, both explicit social criteria and latent textual features were combined to create a comprehensive feature set. This combined approach aimed to harness the strengths of both feature types: the explicit features offering direct social interaction metrics and the latent features providing deeper semantic insights. The same set of classifiers was employed to evaluate this combined feature set.

The evaluation metrics of experiment (1) are presented in Table 13. When utilizing only basic social features such as retweet count, like count, and quote count, GBoost demonstrated the highest accuracy (74.74%), followed closely by other ensemble models. GBoost also achieved the highest F1 score (78.33%) and AUC (81.46%), indicating its strong ability to handle feature variability and complex interactions within the data. However, introducing latent features derived from tweet information, altered the performance dynamics. In this setting, RF showed the highest accuracy (68.50%), reflecting its capacity to effectively leverage complex feature interactions. Notably, the inclusion of latent features resultred in a decrease in performance compared to to the basic features, with the RF algorithm, yielding the best overall results, where all evaluation metrics averaged around 70%.

The combination of both explicit basic and latent features aimed to capitalize on the strengths of feature type. This comprehensive feature set provided a more informative representation of the tweets, leading to improved performance across all classifiers. XGBoost emerged as the top performer, with the highest accuracy (78.43%), F1 score (81.02%), and AUC (86.14%), underscoring its efficiency in handling large, complex datasets with diverse feature sets. Gradient boosting and Random Forest also maintained strong performance, with accuracies of (77.23%) and (78.05%) respectively, further validating their robustness in ensemble learning.

Interestingly, the performance boost was most pronounced in simpler models like LR and SVM, where accuracies improved to (69.17%) and (67.01%) respectively. This suggests that combining explicit and latent features effectively enhanced the predictive power of these models by providing a richer, more nuanced understanding of the data.

Experiment (2): textual features

In the second experiment, the focus was on the textual vectorization of the tweets’ content to assess how different textual representations impact the performance of the classifiers. The same algorithms from Experiment 1 were employed, with the exception of replacing the NBC with multinomial naive Bayes (MNB), which is better suited for text data. Two methods of textual vectorization were utilized. The BoW method represents text by the frequency of words in the corpus without considering word order, transforming each tweet into a vector of word counts. The second method, TF-IDF, is a statistical measure used to evaluate the importance of a word in a document relative to a corpus, mitigating the influence of frequently occurring but less informative words.

The results, as shown in Table 14 indicate that LR achieves the highest overall performance when using BoW, with an accuracy of (91.62%), an F1 score of (92.59%), and an AUC of (97.04%). This suggests that LR effectively balances the trade-off between precision and recall. RF also demonstrates strong performance, with an accuracy of (91.22%) and an F1 score of (92.37%). In contrast, AdaBoost shows the lowest performance among the algorithms, with an accuracy of (73.75%) and an F1 score of (79.60%), highlighting its limited effectiveness in this context. The SVM algorithm performs comparably to LR and RF when using TF-IDF, with an accuracy of (92.13%) and an AUC of (97.26%).

Experiment (3): embedding contextual features

In Experiment (3), AraVec (Soliman, Eissa & El-Beltagy, 2017), a pretrained word representation specifically designed for Arabic text, was employed to embed Arabic tweets. AraVec offers various models tailored to different corpora, and for this study, the Twitter corpus was selected to maintain consistency with the tweet-based dataset. Multiple configurations were explored to assess their impact on the performance of the classification algorithms. These configurations included comparing n-grams with unigrams to capture different levels of contextual information, as well as contrasting the Continuous Bag of Words (CBOW) approach with the Skip-gram model to evaluate their efficacy in capturing semantic relationships. Additionally, two vector sizes 100 and 300 were tested to examine the trade-off between computational efficiency and the richness of word representations. Notably, the Naive Bayes algorithm was excluded from this experiment due to its incompatibility with negative values present in some embedding vectors. These variations were aimed at comprehensively analyzing the influence of different word embedding configurations on the performance of various classification algorithms.

The results in Table 15 shows that KNN consistently outperformed other algorithms, achieving the highest accuracy of (87.63%), an F1 score of (89.20%), and AUC of (93.72%) when using n-grams with Skip-gram at vector size of 300. This suggests that KNN is particularly effective in handling the dense vector representations generated by this configuration, which likely captures more nuanced semantic relationships in the Arabic language data. In contrast, algorithms like AdaBoost generally underperformed, indicating that decision tree-based ensemble methods may struggle with the complexity or sparsity of the vectorized text data. RF and XGBoost also demonstrated strong performance, especially with larger vector sizes, indicating their robustness in handling high-dimensional data. The significant improvement in metrics such as accuracy and F1 when moving from a vector size 100 to 300 in models like RF and XGBoost highlights the importance of vector size in capturing sufficient semantic detail for effective classification.

User veracity assessment

In the context of social networks, where information is shared rapidly and extensively, evaluating the credibility and trustworthiness of individual users becomes crucial (Alrubaian et al., 2021). Veracity assessment involves examining various indicators, such as a user’s social profile (Al-Qurishi et al., 2018; Shu, Wang & Liu, 2018), posting history, network connections (Khan & Lee, 2019), and content characteristics (Abu-Salih et al., 2019). Researchers employ diverse methods, and machine learning algorithms, to assess user veracity (Jia et al., 2019). The VERA-ARAB dataset, which includes over 13,000 distinct users, provides a valuable resource for conducting experiments and constructing models for user veracity assessment within the context of social networks. The breadth and diversity of the user population contained within the dataset present opportunities for robust analysis and the development of effective veracity assessment frameworks.

Tweet topic classification

Tweet topic classification involves categorizing content into predefined classes. In a supervised tweet classification task, the process typically begins with a labeled training set, where each tweet is assigned a specific class, such as “Politics” or “Sports”. The objective is to develop a classification model capable of accurately assigning a class to new tweet texts (Daouadi, Rebaï & Amous, 2021). A key component of such models is the accurate annotation of tweets into relevant classes (Antypas et al., 2022). With seven distinct topic classes in the dataset, researchers are provided with the opportunity to construct and develop more robust models in Arabic language.

Named entity recognition in Arabic tweets

Named entity recognition (NER) is a natural language processing (NLP) task that involves identifying and classifying named entities within text. Named entities refer to specific entities with proper names, such as persons, organizations, locations, dates, and other relevant entities depending on the context (Li et al., 2020). Recent studies have focused on addressing this task in the Arabic language, weather local dialects (Moussa & Mourhir, 2023) or cross-dialects (El Elkhbir et al., 2023). Our dataset contains data about organizations, people, places, and products in across various Arabic dialects, making it a valuable resource for the named entity recognition task.