Leveraging sentiment analysis of food delivery services reviews using deep learning and word embedding

Reviews dataset description

The reviews dataset was introduced by Mustafa et al. (2023). The primary information source we used to gather reviews specific to FDS was the FDS Talabat. Talabat takes online orders from different eateries and offers delivery services to patrons. Customers can review restaurants, drivers, and the app itself through the app’s rating features. Customers can also share their experiences by posting reviews online or by filing complaints. These reviews were gathered both manually and with the aid of an online scraper tool. After that, native Arabic speakers personally labeled each review as positive, negative, or neutral. Only “Arabic” reviews are included in the data collection process at this time. The neutral reviews are very low in the dataset, and this may affect the performance of the classifier towards this class. Figure 2 illustrates the unbalanced multiclass dataset under study. Since Talabat operates throughout a number of Middle Eastern nations, including Jordan, Iraq, and Egypt, as well as Gulf nations like Saudi Arabia and Bahrain, the reviews include the viewpoints of individuals from various backgrounds and dialects, so this will pose a challenge in these reviews.

Additionally, as shown in Fig. 3, the most commonly used terms related to FDS were examined and represented as word clouds, where each word’s size indicates its significance in the text. Figure 3A includes words that appeared in the negative reviews without any preprocessing steps. Figure 3B includes words that appeared in negative reviews after the preprocessing steps were applied. Similarly, Fig. 3C includes words that appeared in the positive reviews without any preprocessing steps. All the words in the positive reviews after applying the preprocessing steps are appeared in Fig. 3D.

Dataset cleaning and NLP steps

Numerous pointless and repetitious reviews are present in the raw datasets, which ought to be effectively removed. Furthermore, people frequently use emojis to express their feelings and sentiments. The cleaning phase includes various automatic and manual steps. The automated multistage cleaning process includes removing duplicated reviews and replacing emojis with appropriate Arabic text that reflects their sentiments (positive, negative, neutral) based on its higher score in Kralj Novak et al. (2015) dictionary.

Arabic language evaluations would contain “Arabizi,” in which Arabic words are written in Latin script (Baly et al., 2017). Franco terms—things like “”, which are represented as “mat3m” are Arabic words written with a combination of English letters and numbers—have thus been swapped with their Arabic synonyms. Since Arabizi reviews and comments are usually considered spam, SA scholars usually remove them from the dataset. We have adopted a different approach for this task: we have developed a rule-based converter that can change reviews from Arabizi to Arabic. The rule-based converter first transforms a review into a list of words or tokens, using punctuation and white spaces as separators. Second, the Arabizi letters are converted to their associated Arabic alternatives using the translations given in Fig. 4 to convert each Arabizi word into an Arabic word, followed by replacing vague words and phrases with clear formal text utilizing our dialects dictionary that was built in the prior work (Mustafa, Khabour & Shatnawi, 2024), and extended in this work by adding more dialects. After handling Arabizi text replacement, the last step is to correct misspellings. Annotating reviews for testing as positive, negative, or neutral is done manually by native Arabic speakers.

Following cleaning, several NLP techniques are applied, starting with normalization, where we apply various normalization strategies. First of all, characters with various forms are converted to a single form, such as the various forms of “alif” as . Dropping diacritics, punctuation, special characters, and digits, as well as stripping tatweel and repeated characters, are also applied. Moreover, the Arabic definite article (the) was eliminated. We implemented the “Removing stopwords” NLP technique utilizing the stopword list of CLTK. We have extended this list to include more missing stopwords. Tokenization of sentences and words is utilized to split the text into separated units. The pyArabic library contains the majority of Arabic normalization approaches (Zerrouki, 2023).

Finally, stemming is used to reduce the text size in a variety of ways. In this stage, we have evaluated the employment of three Arabic stemmers: ISRI, Tashaphyne, and Snowball. ISRI stemmer, developed by Taghva, Elkhoury & Coombs (2005), has put into practice an Arabic root-extraction stemmer that is comparable to the Khoja stemmer but does not have a root dictionary. In addition, if no root was found, the ISRI stemmer returned the normalized form rather than the original, unaltered word. Tashaphyne, supported in Python, provides both stemming and root extraction, unlike the Khoja, ISRI, and Assem. The Snowball stemmer algorithm, also known as the Porter2 stemming algorithm, is an enhanced version of the Porter Stemmer that addresses some of its shortcomings (Porter, 2001). Snowball supports several languages, including Arabic. The Arabic Light Stemmer by Assem is a snowball-based Arabic stemming algorithm (Chelli, Aries & Mennoucho, 2018). It offers only basic text normalization and stemming.

One of the objectives of this study is to measure the performance impact of different stemming strategies, such as light stemmers and root-extraction stemmers. Table 1 provides an example demonstrating the difference between the stemmers based on their strategies to remove appendages such as prefixes and suffixes.

Table 1:

Comparison between Arabic text-stemming techniques.

Text-stemming technique	Example (Arabic)
Original FDS review text
Cleaned Review
Preprocessed review without stemming
Preprocessed review with root stemming by (ISRI)
Preprocessed review with root stemming by (Tashaphyne)
Preprocessed review with light stemming by (Tashaphyne)
Preprocessed review with light stemming by (Snowball)

DOI: 10.7717/peerj-cs.2669/table-1

Features representation and word embedding methods

Word embeddings are vector representations of terms that are distributed; terms with equivalent meanings have similar vector representations. The cosine similarity of the words could be used to calculate their separation. The terms “traveling” and “vacation,” for instance, will be represented by vectors that are closer together. Compared to bigram and trigram techniques, which either ignore or impose associations between phrases or tokens, this is a more descriptive method for conveying text than classic methods like the bag-of-words. Dense vectors make it simpler for deep learning models to converge than sparse ones. Furthermore, the nature of the dataset and the business need are always determining factors. The embedding layer is a dictionary that links integer indices to dense vectors with real values rather than only 0’s and 1’s. The words are dispersed throughout a very high dimensional space, while the vector’s size is fixed. The model receives as input a very big corpus of text, and the words are dispersed into vectors based on that. A variety of methods exist for word embeddings, including GloVe (Pennington, Socher & Manning, 2014), Word2Vec (Mikolov, 2013), and fastText (Bojanowski et al., 2017). Different parameters are taken into account by each technique for representing the words. The input row of the text is an important function in word representation, where word distribution can be altered by word context. In this article, we only looked at three word-embedding methods: fastText (Bojanowski et al., 2017), GloVe (Pennington, Socher & Manning, 2014), and Word2Vec (Mikolov, 2013).

fastText model

Another word embedding method known as fastText was introduced by Bojanowski et al. (2017). The core of it is an unsupervised technique that uses vectors for denoting words. It is an expansion of the Word2Vec framework in which the sub-words are taken into account. It also offers two architectural models: SG and CBOW. But each word in fastText is separated into an n-gram character. Angle brackets are used as a unique boundary to show where a word starts and ends. It is used to distinguish a word from its subword and the word itself. The word “sentiment,” for example, has the fastText representation (sen, sent, enti, ntim, time, ment, ent) when n = 4. The terms sent and time are denoted by the sequences (sent) and (time); they are not the same as the 4-gram sent and time from the original term sentiment. This model can distinguish between a word’s shorter character sequences and prefixes and suffixes by taking sub-words into account. The word itself will be represented in a vector with its set of character n-grams in this model. A hashable list is used to link the sub-words to their parent word, and the vector of the primary word is equal to the sum of the n-gram vectors. FastText works well with infrequent words. Consequently, by breaking a word up into n-grams, its embeddings can still be recovered even if it was not detected during training. In Word2vec or GloVe, respectively, words that are not included in the model vocabulary have no vector representation. This strategy has this important advantage.

A word set of varying lengths makes up each FDS review. To work with integers, a word-to-index dictionary should be constructed. In the word-to-index dictionaries, every term in the corpus functions as a key, and the value of each key is determined by its associated unique index. In this stage, each word in the review is replaced by its corresponding word embedding, which can be either one of two types: pre-trained distributed word representations or one that is specifically built for the review. BOW, Skip-Gram, fastText, and GloVe are the four different embeddings that we use. In this study, the AraVec 3.0 (Soliman, Eissa & El-Beltagy, 2017) project was used to produce unigram models of the Arabic Skip-gram and CBOW models. Both models were developed using tweets that were taken directly from Twitter. Each model contains 1,259,756 vocabularies, and each model’s vector size is 300. The corpus employed in this study’s Arabic GloVe model is made up of 1.75 billion tokens, 1.5 million vocabulary words, and 256 vector sizes. The Arabic language’s standardized fastText word vectors were pre-trained using the Common Crawl and Wikipedia resources. For the training, a position weight with a dimension of 300 and a continuous bag of words (CBOW) was employed. According to the fastText library (Facebook, 2024), the character n-grams have a length of five characters. The details of the chosen CBOW, Skip-Gram, GloVe and fastText models are provided in Table 2.

Table 2:

Description of the examined pre-trained Arabic words embedding.

Pre-trained word embedding model	Model corpus domain	Number of documents	Number of vocabularies	Vector dimension
CBOW	Twitter (unigrams)	66,900.000	1,259,756	300
Skip-Gram	Twitter (unigrams)	66,900.000	1,259,756	300
GloVe	Multiple resources such as Arabic tweets and Wikipedia	————-	1,500,000 (1.75 billion) tokens	256
FastText	Common crawl (Wikipedia)	—————	——————–	300

DOI: 10.7717/peerj-cs.2669/table-2

Each text review is displayed in the aforementioned model as a 2D vector with a dimension of n $\times$ d, where n is the term count in the text review and d is the vector representation of a word’s dimension length. We have assessed different vector dimension lengths, including d = 50, 64, 100, 256, and 300. We adopt the padding strategy, to pad the representation of each review with zeros, to guarantee that all reviews are the same fixed size. In this manner, each review will have a size of n′ $\times$ d, where n′ was set at 100 for all experiments. Table 3 demonstrates how to convert each FDS review to a vector representation using word embedding.

Table 3:

Example of converting textual review to a vector representation using word embedding.

Text-stemming step	Example (Arabic)
Origin review text
Tokenized review after cleaning, preprocessing, and stemming
Text indexing (encoding)	[2, 3, 14, 161, 304, 70, 43, 40, 1, 82, 71, 63, 3,148]
Padding	[2, 3, 14, 161, 304, 70, 43, 40, 1, 82, 71, 63, 3,148,….., 0,0,0,0,0,0,0,0,0,0]
Vector representation using word embedding	[[5.222600e−02, 4.599880e−01, −2.192200e−01, −1.810400e−02, 4.585380e−01, 2.424030e−01, 1.037320e−01, 4.975420e−01, −8.883430e−01, −4.546760e−01, 8.732000e−02, 3.694500e−02,…], [−3.264180e−01, −2.968420e−01, 5.262500e−02, 3.534710e−01, −7.600600e−01, −6.230630e−01, 6.521930e−01, 3.025940e−01,…], [−1.751270e−01, 2.583510e−01, −2.118470e−01, 4.183250e−01, 1.825560e−01, −3.380570e−01, −3.754840e−01, −5.319610e−01,…], …….. ]]

DOI: 10.7717/peerj-cs.2669/table-3

Sentiment classification and deep learning models

Deep learning has been frequently employed by authors in the field of SA, where it demonstrated encouraging outcomes and effective performance (Ain et al., 2017). Using massive datasets, deep learning methodology is applied to build high-level model abstractions and nonlinear transformations. Typically, deep learning text categorization model architectures are made up of the following elements connected in a particular order as depicted in Fig. 5:

• -

  Embedding layer: keeps track of a lookup table that transforms words with numerical indexes into their dense vector representations.

• -

  Deep network: The deep network receives the embedding vectors and transforms them into a compressed representation. The compressed representation fully captures the information included in the text’s word sequences. RNNs or variations of them, such as LSTM/GRU, typically make up the deep neural network component. To combat the propensity for overfitting, the dropout layer is added.

• -

  Fully connected layer: converts the RNN/LSTM/GRU’s deep representation into the final output classes or class scores. This component is made up of fully linked layers, batch normalization, and optional dropout layers for regularization.

• -

  Output layer: this layer produces either Sigmoid for binary classification or Softmax for both binary or multi-classification, depending on the problem at hand.

Our experimentations considered four groups to test sentiment classification: (1) using WE which is learned from scratch. (2) Using static pre-trained WE. (3) Using non-static pre-trained WE, and (4) using specially generated WE. Furthermore, experiments utilized three DNN topologies: CNN, BiLSTM, and a hybrid LSTM-CNN.

Convolutional neural network model

CNN has been successfully used to solve challenges involving document classification and natural language processing (Albawi, Mohammed & Al-Zawi, 2017). The proposed CNN model has an embedding layer with the following settings:

• embedding-size = 50, 64, 100, 256, and 300 (depends on the used WE method)

• max-len set to “100” and max-features set to num-unique-word (depends on the used stemmer)

• 5-kernel, rectified linear (or “relu”) activation function is utilized with 200 filters (parallel fields for word processing) in a conservative CNN configuration dropout = 0.3

• MaxPooling1D with pool-size = 5. The pooling layers can be viewed as a method of downsampling (decreasing the size of) the input feature vectors. The dropout layer is one of the most effective and commonly used regularization techniques in NNs.

Then, the model utilized another convolution layer with 200 filters, having a 5-kernel, rectified linear (or “relu”) activation function. GlobalMaxPooling1D layer is added here. Whereas MaxPooling1D or GlobalMaxPolling1D layers’ primary responsibility is to lessen the dimensionality of each convolutional output, Conv1D layers are responsible for computing the convolution operations. The MaxPooling window then outputs patches of maximum values by extracting the largest value contained inside following the convolution operation. A 32-dense layer with the sigmoid activation function is used to empower the network’s ability to classify the extracted features better, followed by another dropout layer of 0.3. The output layer outputs a value of 0, 1, or 2 for the review’s neutral, positive, and negative feelings, respectively, using a softmax activation function. A dense layer with three output units representing the three potential classes of our dataset is being stacked. ’Softmax’ is an excellent activation function to use in the final layer if producing probabilistic outputs. Figure 6 shows the overall CNN model layers architecture in the specified order.

BiLSTM model

Hochreiter (1997) describes LSTM as a unique type of RNN that can recognize long-term dependencies. The long-term dependency issue that the conventional RNN suffered is specifically avoided by LSTM architecture. These networks are used for recurring objects, such as time series data. They can understand the connections between the components of an input sequence. In numerous text mining and English SA projects, they showed excellent accuracy (Tai, Socher & Manning, 2015). Unlike the LSTM, which reads the sequence from left to right, the bidirectional LSTM (Bi-LSTM) combines two layers that are moving in contrary directions (forward and backward) over the same output. The output layer concurrently accepts data from the prior sequence (backward) and subsequent sequence (ahead) states. As a result, it is very helpful in situations when the context of the input is needed, for instance when the negation phrase comes after a positive word (Schmidhuber, 2015). Our BiLSTM model includes an embedding layer with the following settings: embedding-size = 50, 64, 100, 256, and 300 (depends on the used WE method), max-len set to “100” and max-features set to num-unique-word (depends on the used stemmer). The following layer includes 64 units of BiLSTM layer followed by a fully connected sigmoid activation layer with 32 output units and a 0.3 dropout layer. Ultimately, a dense softmax layer that forecasts the three sentiment class labels is combined. Figure 7 shows the overall BiLSTM model layers architecture in the specified order.

LSTM-CNN model

We proposed a hybrid LSTM-CNN model that combines many layers together for better sentiment classification results. This model includes an embedding layer with similar settings as in the CNN and BiLSTM models: embedding-size = 50, 64, 100, 256, and 300 (depending on the used WE method), max-len set to “100” and max-features set to num-unique-word (depends on the used stemmer). The next stage consists of two LSTM layers with 200 and 64 output units respectively, followed by a convolutional layer with 200 filters, and a kernel of size 5. A Rectified Linear Unit (ReLU) activation function is used due to its effectiveness and low computational complexity. To downsample feature maps produced by the convolutional layers, MaxPooling1D layer with a pool-size of 5 is added along with a dropout layer set to 0.3. The same three previous consecutive layers are repeated (the Conv1D, MaxPooling1D, and dropout) with the same settings. We applied a 32-dense layer with sigmoid activation to a fully connected layer after flattening the results into a single vector. Finally, a 3-dense with a softmax function that forecasts the three labels is combined. Figure 8 summarizes the overall LSTM-CNN model layers architecture in the specified order.

Experiments setup

The Talabat online food delivery service (FDS) is the subject of the experimentation dataset, which is linked to a raised concern in the COVID-19 emerging catastrophe. The original collection and use of the dataset were made by Mustafa et al. (2023). The users decided to share their thoughts on this matter on the Talabat website and application. The web scraper program was used to gather the dataset between October 1, 2021, and December 3, 2021. Table 4 contains information about the dataset statistics.

The three deep learning models will be used in the experimental setup to create and evaluate models in a Python 3.7.6 environment for text review classification on identical hardware and operating systems. All experiments have the following configurations:

• -

  Operating system: Windows 11 Education 64-bit (10.0, Build 18363)

• -

  System manufacture model: HP Inc. Precision 5720 AIO

• -

  Processor: Intel(R) Xeon(R) CPU E3-1275 v6 @ 3.80 GHz (8 CPUs), ~3.8 GHz

• -

  Memory: 16,384 MB RAM (16 GB)

• -

  GPU: Radeon (TM) Pro WX 7100 Graphics.

Table 5 lists several library functions that were used.

The parameters for embedding can be either specified as the length of the vector, while others can be trained as weights. In the training phase, weights are learned by the model and for the dense layer too. Related groups of words that are closer together can be found in the corresponding embedded vectors (Al-Bayati, Al-Araji & Ameen, 2020). Word embeddings can be learned using a variety of techniques, including Word2vec (Google), GloVe (Stanford), FastText (Facebook), and the Embedding layer (Keras). The embedding layer is the initial hidden layer of the model that prepares data to be fed as input for the subsequent layer of the model. In this work, every one of the four learning techniques has been tested and used.

The Sequential model is a stack of linear layers, it offers the ability to utilize any of the many Keras-compatible layers. The Dense layer is the standard densely linked neural network layer with all the weights and biases. The Sequential model can be constructed by adding layers one at a time. Additionally, including a list of indicators that can be utilized for future evaluation but have no impact on training. In this case, we want to employ the Adaptive Moment Estimation (Adam) optimizer (Kingma & Ba, 2015) along with the categorical cross entropy as the loss function, which is suitable for multi-class classification. The dataset is split into two parts: 90% for training, and 10% for testing. During the training process, 10% of the training data were utilized as validation data. Additionally, we performed five-fold cross-validation by averaging the results of five distinct runs for each data division, which yields enhanced performance. After training is completed, the embedding layer learns the weights, which are simply the vector representations of each word. The goal of separating testing from the validation is to select the model with the highest validation accuracy and then test it with the testing set. We must define three parameters for the input layer: the vocabulary size, the embedding dimension, and the sequence length. As the stemmer algorithm changes, so does the vocabulary size. Table 6 displays the various vocabulary sizes for each stemmer employed. Table 6 estimates how many of each vocabulary term was discovered in each pre-trained WE model. The no-stemming technique provided the bigger vocabulary for the FDS dataset, as it includes (28,035) words, whereas Tashaphyne’s root stemming strategy generates the least vocabulary, as it comprises (7,420) words. Also, utilizing GloVe WE always delivers greater coverage across all embedding approaches.

It is necessary to provide the number of iterations that the model will go through during training. The completed iterations are commonly referred to as epochs. For all experiments, the epochs are initially set to 20. The Early Stopping observed validation loss during the training process, which was used to avoid over-fitting and automatically alter the number of epochs. If the Early Stopping determined that the validation loss rose for three iterations (patience = 3) during training, the training model would be terminated. The batch size determines how many samples are utilized in one epoch or how many samples are used in one forward/backward pass. We changed the size to 16, 32, and 64 each time. The batch size 64 gives us the greatest results in our trials by tracking accuracy and loss, which can suggest overfitting when the training accuracy increases and the validation accuracy decreases. It might be argued that the large batch size has a negative impact on validation accuracy, convergence speed, and execution time. Many researchers recommend using smaller batch sizes (fewer than 64) to improve performance. The learning rate was set to 0.001. This rate governs the mechanism for updating the weights when each batch is completed. After conducting numerous trials with different hyperparameters using the trial-and-error method, the hyperparameters utilized in the basic models were chosen. To determine the ideal hyperparameter values, exhaustive experiments were carried out; Table 7 displays the hyperparameters that produced the best results for the model.

Evaluation metrics

Recall (R) indicates a classifier’s completeness, while precision (P) evaluates a classifier’s exactness. P and R can be combined to provide the single metric known as F1, sometimes known as the F-score, which is the weighted harmonic mean of both measures. The weighted accuracy is calculated as well to evaluate our model. The calculations of R and P are shown in Eqs. (7) and (8):

(7) $$R=\frac{TP}{TP+FN}$$

(8) $$P=\frac{TP}{TP+FP}.$$

True positive (TP) defines how often the model categorizes a positive sample as positive correctly. False negative (FN) defines how often the model misclassifies positive samples as negative. False positive (FP) defines how often the model misclassifies a negative sample as a positive one. True negative (TN) defines how often a negative sample gets the right classification from the model. For multi-class classification, we want to find TP, FN, FP, and TN for each class separately. For example, in Fig. 9 which presents a 3-class confusion matrix, let’s designate the target to be the C1 class. All other classes have negative classifications, while this one has a positive classification. Then Fig. 9 demonstrates how to determine each of them for class C1.

The difference between the issued real value and the model’s prediction is referred to as loss. The greater the loss, the less accurate the model. A categorical cross-entropy function is used to determine the loss for multi-class classification. As a result, loss evaluation is depicted in Eq. (9) (Grandini, Bagli & Visani, 2020).

(9) $$Loss=-\sum _{i=1}^s{y}_i\times \log \widehat{{\mathbf{y}}_{\mathbf{i}}}$$where $y_i$ is the goal value, $\widehat{{\mathbf{y}}_{\mathbf{i}}}$ is the $i^{th}$ scalar value in the model output, and $s$ is the number of scalar values in the model output.

Results and discussion

We carried out extensive research to evaluate the various elements for deals with word embeddings such as the effect of using the various stemmers to create the FDS reviews corpus vocabulary, the various weighting vectors methods, random, static, non-static, and custom, and the effect of embedding vectors dimension size. Each method has its advantages and disadvantages over the other.

Experiments (A): learning embedding from scratch

During the first group experiments, we examined learning a word embedding as part of fitting a neural network model, where random weights are used to initialize the word vectors, and then they are updated during the training process using a backpropagation method. We also evaluate the impact of various stemming techniques across various embedding dimension sizes. The results of ASA employing random weight initialization approaches in conjunction with deep learning methods are displayed in Fig. 10. This includes CNN-rand, BiLSTM-rand, and LSTM-CNN-rand results from 3-Class Sentiment Classification Using Deep Learning. We started from scratch when training the embedding layer, using four different stemmers (ISRI, stem-based Tashaphyne, root-based Tashaphyne, and Snowball) and dimension sizes (50, 64, 100, and 300).

Learning embedding from scratch is generally not particularly precise when dealing with sequential data. Techniques that take into account local and sequential data rather than absolute positional data are the focus when working with sequential data. This is demonstrable through performance. Using Tashaphyne-based roots and the BiLSTM classifier model, the accuracy results achieve the highest value of 83.63% when the embedding dimension is adjusted to 300, as per the scratch-learned embedding technique. For the same experiment’s setting, the loss scores the lowest as 44.76%. When utilizing one of the stemming techniques—root stemming, ISRI, or Tashaphyne in scratch embedding—generally yields the best results. The highest accuracy of 82.72% is achieved with ISRI.

Experiments (B): static pre-trained word embeddings

It can take a while for the network to learn a word embedding, especially for very large text datasets. In this section, we will use pre-trained word embedding developed on a large text corpus. The first step is to load the word embedding as a word dictionary into vectors. The word vectors are positioned correctly in the weight matrix, which is built using the loaded embedding and the tokenizer word-index vocabulary as inputs.

In order to prevent the network from attempting to modify the pre-learned vectors as part of network training, observe that we set the ‘trainable’ argument to ‘False’ and provide the previously created weight matrix embedding-vectors as an argument to the fresh embedding layer. Figure 11 shows the results of ASA using deep learning methods with static pre-trained weights.

According to the results of the static weights experiments, using a skip-gram with a snowball stemmer gives us the best accuracy at 83.6% when the LSTM-CNN classifier model is used. However, when the used classifier model is the standalone CNN, the original text without stemming worked well and was similar to Snowball light stemming results after we combined it with GloVe or fastText word embedding, where the accuracy scores were 82.4%. Using the BiLSTM classifier, the best accuracy was using Snowball or original text at 83.57%, and the lowest loss value was 44.93% with the GloVe.

Experiments (C): non-static pre-trained word embeddings

In this group of experiments, the weights in the embedding layer can serve as a network’s initial start and be modified as the network is trained. ‘trainable = True’ (the default parameter) can be used to achieve this when creating the embedding layer. Figure 12 shows the results of ASA using deep learning methods with the non-static pre-trained weights technique.

Using Tashaphyne stems with CBOW embedding yields the best accuracy results for BiLSTM and LSTM-CNN models, at 83.44% and 84%, respectively, based on the findings of non-static weights. The BiLSTM model scores the lowest loss values with the fastText word embedding as 44.93%. Using the snowball stemmer and skip-gram with BiLSTM produced the best recall rates across all group experiments, reaching 84.38%.

Experiments (D): custom word embedding

Through this group experiments, we demonstrate how effectively we can acquire a standalone embedding that we can then apply to our neural network. The Word2Vec algorithm is a method for learning word embedding independently from a text collection. The method’s advantage is that it is capable of efficiently constructing high-quality word embedding concerning space as well as time complexity. First, the list of cleaned and preprocessed reviews from the training data is passed in. Then, we designated the dimensions of the embedding vector space to be (300). We take into account five neighbors when determining how to embed each word in the training reviews. Four threads were used during model fitting. We set the lowest count of occurrences for words to take into account in the vocabulary to one as we are learning new words. Figure 13 shows the results of ASA using deep learning methods with custom embedding weights technique.

The custom embedding technique indicates that we get superior accuracy results than the CBOW embedding when we use skip-gram with Tashaphyne-based roots. Specifically, we achieve 83.08%, 83.28%, and 83.05% using CNN, BiLSTM, and LSTM-CNN, respectively. The BiLSTM model here scores the lowest and the best loss value as 44.38%. We can notice that the BiLSTM model with Tashaphyne roots always performs accurately according to all group experiments in terms of loss value. Trainable outperforms scratch by 1.34 3.4%. When trainable = true is used, fastText accuracy decreases without utilizing a stemmer enhancement. CNN skip-gram used the Tashaphyne stemmer to obtain the best improvement, which was 3.5%. With ISRI stemmer improvement, trainable = true exhibits an accuracy improvement of up to 3%.

The accuracy results of the best experiment conducted by each group, scratch, static weights, non-static weights, and custom weights are shown side by side in Fig. 14. According to the results, CNN-LSTM had the lowest accuracy (79%) when utilizing bespoke weights without a stemmer, and the highest accuracy (84%) while using the Tashaphyne stemmer with non-static weights.

External evaluation and comparison

We used benchmark datasets, which were also used by other academics to evaluate their models, to compare our model with those of other researchers. Two sizable collections of reviews of books and hotels written in dialectical Arabic and Modern Standard Arabic (MSA) are called Hotels Arabic-Reviews Dataset (HARD) (Elnagar, Khalifa & Einea, 2018) and Books Reviews in Arabic Dataset (BRAD) (Elnagar & Einea, 2016). There are reviews and ratings out of five for each of them. By classifying ratings 1 and 2 as negative (−1), ratings 3 as neutral (0), and ratings 4 and 5 as positive (+1), we may convert the 5-star rating to 3-polarity scaling. Following rating conversion, there are 409,562 reviews total on HARD, divided into 276,387 positive, 52,849 negative, and 80,326 neutral reviews. There are 325,434 positive reviews, 78,380 negative reviews, and 106,785 neutral reviews out of the 510,599 total reviews on BRAD.

The confusion matrix presented in Fig. 15 shows the performance of the best model (LSTM-CNN) for multiclass classification. The normalized matrix evaluates the true positive rates towards each separated class. The area under the curve (AUC) is a single number that sums up a classifier’s performance by evaluating the ranking regarding the separation of each class as well. ROC is a probability curve that illustrates the degree or measure of separability by mapping the true positive rate (TPR), which represents the sensitivity vs. false positive rate (FPR), which represents specificity at various threshold levels. The receiver operating characteristic (ROC) is an excellent tool to visualize a classifier performance (Brownlee, 2023). Better still, the higher. The area under the ROC curve (AUC), which ranges from 0 to 1, represents the percentage of this region. The misclassification of neutral sentiments as positive or negative is a common issue in sentiment analysis models, especially in Arabic reviews. This is due to complex sentence structures and ambiguous language, which models struggle to interpret effectively. From both Figs. 15 and 16 we can see that the model operates worst with the neutral class, where only 13% of the actual neutral reviews were truly classified, while the remaining reviews were misclassified as either positive or negative. For example, the review was accidentally misclassified as positive; this may happen since it contains a positive word as .

Figure 17 shows the normalized confusion matrix of LSTM-CNN for binary classification, Fig. 18 explains that the ROC curve becomes better when neutral reviews are excluded from the dataset, where the average AUC increases from 0.77 to 0.9.

To establish a binary sentiment classification, neutral reviews were removed from the subsequent studies. We determined which experimentation settings worked best on the FDS and we found that skip-gram word embedding and stemming based on the Tashaphyne root produced the best results most of the time. Table 8 displays the results of the performance of each experiment.

The results we achieved were compared to those of a few other state-of-the-art deep learning models found in the literature. For Arabic SA, Nassif, Darya & Elnagar (2021) suggested Deep and Shallow Learning approaches. All conceivable hybrids of the three primary DL archetypes—CNN, LSTM, and GRU—, as well as all other archetypes, perform binary and 5-class classification. Their results when running binary classification reached: 94.2% and 83.5% respectively on HARD and BRAD.

The same corpora were utilized by Berrimi et al. (2023) to evaluate their hybrid bidirectional GRU/LSTM model for binary classification. They scored promising accuracies of 96.29% and 95.65% respectively on both datasets. Muaad et al. (2021) proposed ArCAR, a novel computer-aided deep-learning Arabic text recognition system that can analyze and interpret Arabic text at the character level. Each Arabic character in the Arabic text input is quantized into a 1D vector to create a 2D array for the ArCAR system. Five-fold cross-validation tests have approved the ArCAR system for two usages, Arabic text document categorization and Arabic sentiment analysis. The proposed deep learning ArCAR consists of two fully connected or dense layers and six convolutional layers for Arabic sentiment analysis. The model achieved the highest performance using the Arabic hotel reviews dataset (HARD) of two classes in terms of overall accuracy of 93.58%. The model employed the HARD and BRAD datasets for binary and 3-class sentiment classification. Additional studies that used the HARD and BRAD datasets provided by Habbat et al. (2023) for 3-class classification. They used a stacking deep learning model for classification, AraBERT for text representation, and AraGPT for data augmentation. Three deep learning models—GRU, LSTM, and Bidirectional LSTM (BiLSTM)—serve as the foundation classifiers for the suggested ensemble model. The accuracy for HARD and BRAD was 88.89% and 90.88%, respectively.

Table 9 summarizes the results of several state-of-the-art deep learning models compared to our proposed model. Comparing our results with previously described deep learning models, we scored the best binary classification results on HARD at 97.2%; our results on BRAD are comparable and close to some other models, such as the study of Nassif, Darya & Elnagar (2021). By comparing three classification outcomes, our results are also comparable and close to the model of Habbat et al. (2023) on the HARD dataset and are higher than the accuracy results of Muaad et al. (2021) on BRAD by approximately 4%.