Classification of movie reviews using term frequency-inverse document frequency and optimized machine learning algorithms

Data description

This study uses the ‘IMDb Reviews’ from Kaggle which contains users’ reviews about movies (DAT, 2018). The dataset has been largely used for text mining and consists of reviews of 50,000 movie reviews of which approximately 25,000 reviews belong to the positive and negative classes, respectively. Table 2 shows samples of reviews from both negative and positive classes.

TextBlob

TextBlob is a Python library that we used to annotate the dataset with new sentiments (Tex, 2020; Loria, 2018). TextBlob is used for labeling as the possibility of contradiction between the review text and label can not be ignored. TextBlob finds the polarity score for each word and then sums up these polarity scores to find the sentiment. TextBlob assigns a polarity score between −1 and 1. A polarity score greater than 0 shows the positive sentiment, a polarity score less than 0 shows a negative sentiment while a 0 score indicates that the sentiment is neutral. In the dataset used in this study, 23 neutral sentiments are found after applying TextBlob. Pertaining to the low number of neutral sentiments which can cause class imbalance, only negative and positive sentiments are used for experiments. Contradiction in TextBlob annotated label and original dataset label is shown in Table 3.

Feature engineering methods

Identification of useful features from the data is an important step for the better training of machine learning classifiers. The formation of secondary features from the original features enhances the efficiency of machine learning algorithms (Oghina et al., 2012; Mujahid et al., 2021). It is one of the critical factors to increase the accuracy of the learning algorithm and boost its performance. The desired accuracy can be achieved by excluding the meaningless and redundant data. Less quantity of meaningful data is better than having a large quantity of meaningless data (Prabowo & Thelwall, 2009). So, feature engineering is the process of extracting meaningful features from raw data which helps in the learning process of algorithms and increases its efficiency and consistency (Lee, Cui & Kim, 2016).

Term frequency-inverse document frequency

TF-IDF is another feature engineering method that is used to extract features from raw data. It is mostly deployed in areas like text analysis and music information retrieval (Yu, 2008). In this approach, weights are assigned to every term in a document based on term frequency and inverse document frequency (Neethu & Rajasree, 2013; Biau & Scornet, 2016). Terms having higher weights are supposed to be more important than terms having lower weights. The weight for each term is based on the Eq. (1).

(1) $$W_{i,j}=T{F}_{t,d}\left({\displaystyle \frac{N}{D_t}}\right)$$where TF_t,d is the number of occurrences of term t in document d, D_f,t is the number of documents having the term t and N is the total number of documents in the dataset.

TF-IDF is a kind of scoring measurement approach which is widely used in summarization and information retrieval. TF calculates the frequency of a token and gives higher importance to more common tokens in a given document (Vishwanathan & Murty, 2002). On the other hand, IDF calculates the tokens which are rare in a corpus. In this way, if uncommon words appear in more than one document, they are considered meaningful and important. In a set of documents D, IDF weighs a token x using the Eq. (2).

(2) $$IDF(x)=N/n(x)$$where n(x) denotes frequency of x in D and N/n(x) denotes the inverse frequency. TF-IDF is calculated using TF and IDF as shown in Eq. (3).

(3) $$TF-IDF=TF\times IDF$$

TF-IDF is applied to calculate the weights of important terms and the final output of TF-IDF is in the form of a weight matrix. Values gradually increase to the count in TF-IDF but are balanced with the frequency of the word in dataset (Zhang et al., 2008).

Word2Vec

Word2Vec is one of the widely used NLP techniques for feature extraction in text mining that transforms text words into vectors (Wang, Ma & Zhang, 2016). Given a corpus of text, Word2Vec uses a neural network model for learning word associations. Each unique word has an associated list of numbers called ‘vector’. The cosine similarity of the vectors represents the semantic similarity between the words that are represented by vectors.

GloVe

GloVe from Global Vectors is an unsupervised model used to obtain words’ vector representation (Bhoir, Ghorpade & Mane, 2017). The vector representation is obtained by mapping the words in a space such that the distance between the words represents the semantic similarity. Developed at Stanford, GloVe can determine the similarity between words and arrange them in the vectors. The output matrix by the GloVe gives vector space of word with linear substructure.

Supervised machine learning models

Several machine learning classifiers have been selected for evaluating the classification performs in this study. A brief description of each of these classifiers is provided in the following sections.

Proposed methodology

With the growing production of movies over the last two decades, a large number of opinions and reviews are posted on various social media platforms and websites. Such reviews are texts that show explicit opinions about a film or product. These opinions play an important part in the success of film or sales of the products (Agarwal & Mittal, 2016). People search blogs, and evaluation sites like IMDb to get the likes and dislikes of other people about films, the cast, and team, etc. but it is very difficult to read every review and comment. Evaluation of these sentiments becomes beneficial to assisting people in this task. Sentiments expressed in such reviews are important regarding the evaluation of the movies and their crew. Automatic sentiment analysis with higher accuracy is extremely important in this regard and this study follows the same direction and proposes an approach to perform the sentiment analysis of movie reviews. In addition, since the contradictions in the expressed sentiments in movie reviews and their assigned labels can not be ignored, this study additionally uses TextBlob to determine the sentiments. Two sets of experiments are performed using the standard dataset and TextBlob annotated dataset to fill in the research gap as previous studies do not consider the contradictions in the sentiments and assigned labels. Figure 1 shows the flow of the steps carried out for sentiment classification.

As a first step, the reviews are preprocessed using a sequence of operations. Preprocessing is critical to boosting the training of the classifiers and enhancing their performance. The purpose of preprocessing is to clean the data by removing unnecessary, meaningless, and redundant text from reviews. For this purpose, the six steps are carried sequentially, as shown in Fig. 2.

Punctuation is removed from IMDb text reviews because punctuation does not add any value to text analysis (Guzman & Maalej, 2014). Sentences are more readable for humans due to punctuation, however, it is difficult for a machine to distinguish punctuation from other characters. Punctuation distorts a model’s efficiency to distinguish between entropy, punctuation, and other characters (Rupapara et al., 2021b, Liu et al., 2008). Punctuation is removed from the text in pre-processing to reduce the complexity of the feature space. Table 5 shows the text of a sample review, before and after the punctuation has been removed.

Once the punctuation is removed, the next step is to find numerical values and remove them as they are not valuable for text analysis. Numerical values are used in the reviews as an alternative to various English words to reduce the length of reviews and ease of writing the review. For example, 2 is used for ‘to’ and numerical values are used instead of counting like 1 instead of ‘one’. Such numerals are convenient for humans to interpret, yet offer no help in the training of machine learning classifiers. Table 6 shows text from sample reviews after the numeric values are removed.

In the subsequent step of numbers removal, all capital letters are converted to lower form. Machine learning classifiers can not distinguish between lower and upper case letters and consider them as different letters. For example, ‘Health’, and ‘health’ are considered as two separate words if conversion is not performed from uppercase to lowercase. This may reduce the significance of most occurred terms and degrade the performance (Liu & et, 2010). It increases the complexity of the feature space and reduces the performance of classifiers; therefore, converting the upper case letters to lower form helps in increasing the training efficiency of the classifiers. Table 7 shows the text after the case is changed for the reviews.

Stemming is an important step in pre-processing because eliminating affixes from words and changing them into their root form is very helpful to enhance the efficiency of a model (Goel, Gautam & Kumar, 2016). For example, ‘help’, ‘helped’, and ‘helping’ are altered forms of ‘help’, however, machine learning classifiers consider them as two different words (Singh et al., 2013b). Stemming changes these different forms of words into their root form. Stemming is implemented using the PorterStemmer library of Python (Pang, Lee & Vaithyanathan, 2002). Table 8 shows the sample text of review before and after stemming.

The last step in the preprocessing phase is the removal of stop words. Stop words have no importance concerning the training of the classifiers. Instead, they increase the feature vector size and reduce the performance. So they must be removed to decrease the complexity of feature space and boost the training of classifiers. Table 9 shows the text of the sample review after the stopwords have been removed.

After the preprocessing is complete, feature extraction takes place where BoW, TF-IDF, and GloVe are used. Feature space for the sample reviews is given in Tables 10 and 11 for BoW and TF-IDF features, respectively. Experiments are performed with the standard dataset, as well as, the TextBlob annotated dataset to analyze the performance of the machine learning and proposed models.

The data are split into training and testing sets in a 75 to 25 ratio. Machine learning classifiers are trained on the training set while the test set is used to evaluate the performance of the trained models. For evaluating the performance, standard well-known parameters are used such as accuracy, precision, recall, and F1 score.

Evaluation parameters

Performance evaluation of the classifiers requires evaluation metrics for which accuracy, precision, recall, and F1 score are selected concerning their wide use. The introduction of the confusion matrix is necessary to define the mathematical formulas for these evaluation metrics. The confusion matrix as shown in Fig. 3 can be considered as an error matrix that indicates four quantities. The confusion matrix shows true positive (TP), false positive (FP), true negative (TN), and false-negative (FN). Each row of the matrix represents the actual labels while each column represents predicted labels (Landy & Szalay, 1993).

TP indicates that the classifier predicted the review as positive and the original label is also positive. A review is TN if it belongs to the negative class and the real outcome is also negative. In the FP case, the review is predicted as positive, but the original label is negative. Similarly, a review is called FN if it belongs to the positive class but the classifier predicted it as negative (Rokach & Maimon, 2005).

Accuracy is a widely used evaluation metrics and indicates the ratio of true predictions to the total predictions. It has a maximum value of 1 for 100% correct prediction and the lowest value of 0 for 0% prediction. Accuracy can be defined as

(7) $$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

Precision focuses on the accuracy of predicting the positive cases. It shows what proportion of the positively predicted cases is originally positive. It is defined as

(8) $$Precision={\displaystyle \frac{TP}{TP+FP}}$$

Recall calculates the ratio of correct positive cases to the total positive cases. To get the ratio, the total number of TP is divided by the sum of TP and FN as follows

(9) $$Recall={\displaystyle \frac{TP}{TP+FN}}$$

F1 score is considered an important parameter to evaluate the performance of a classifier and has been regarded as more important than precision and recall. It defines how precise and robust is the classifier by incorporating precision and recall (Bruce, Koger & Li, 2002). F1 score value varies between 0 and 1 where 1 shows the perfect performance of the classifier. F1 score is defined as

(10) $$\mathrm{F}1Score=2\times {\displaystyle \frac{precision\times recall}{precision+recall}}$$

Results using BoW features

Table 12 shows the classification accuracy of the machine learning classifiers when BoW features to train and test the classifiers. Results indicate that SVM can achieve an accuracy of 0.87 with BoW features. Overall, the performance of all the classifiers is good except for DT whose accuracy is 0.72.

Performance of the classifiers is given in Table 13 in terms of precision, recall, and F1 score. The F1 score indicates that its value is the same with both positive and negative classes for all the classifiers, except for GBC who has F1 scores of 0.86 and 0.85 for positive and negative classes, respectively. Precision values are slightly different for positive and negative classes; for example, SVM has a precision of 0.88 and 0.90 for positive and negative classes. Similarly, although precision, recall, and F1 score of DT are the lowest but the values for positive and negative classes are almost the same. An equal number of the training samples in the dataset makes a good fit for the classifiers, and their accuracy and F1 scores are in agreement.

Results using TF-IDF features

Table 14 contains the accuracy results for the classifiers using the TF-IDF features. It shows that the performance of the SVM has been elevated with an accuracy of 0.89 which is 2.29% higher than that of using BoW features. Unlike BoW which counts only the frequency of terms, TF-IDF also records the importance of terms by assigning higher weights to rare terms. So, the performance is improved when TF-IDF features are used as compared to BoW features.

Results for precision, recall, and F1 score are given in Table 15. Experimental results indicate that the F1 score is the same for positive and negative classes for all classifiers which indicates the good fit of the modes on the training data. On the other hand, precision for positive and negative classes is slightly different. For example, GBS has a precision of 0.84 and 0.87 while SVM has a precision of 0.88 and 0.90 for positive and negative classes, respectively.

SVM performs better for text classification than other supervised learning models, especially in the case of large datasets as this algorithm is derived from the theory of structural risk minimization (Mouthami, Devi & Bhaskaran, 2013).

Classifiers results using GloVe features

Experimental results using GloVe features are shown in Table 16 for the selected classifiers. Results suggest that the performance of all the classifiers has been degraded when trained and tested on GloVe features. Glove features are based on the global word-to-word co-occurrence and count the co-occurred terms from the entire corpus. GloVe model is traditionally used with deep learning models where it helps to better recognize the relationships between the given samples of the dataset. In machine learning models, its performance is poor than that of TF-IDF features (Dessi et al., 2020). SVM and RF outperform other models using GloVE features.

Results using Word2Vec features

Performance elevation metrics for all the classifiers using the Word2Vec features are given in Table 17. Results indicate that the performance of the classifiers is somehow better when trained and tested on Word2Vec features in comparison with GloVe features results. The performance of classifiers is not significant using Word2Vec features in comparison to the results of the classifiers using BoW and TF-IDF features. SVM achieved the highest accuracy of 0.88 with Word2Vec features as compared to other models because Word2Vec gives linear features set which is more suitable for SVM as compared to RF, GBC, and DT.

The comparison between machine learning models results on original dataset sentiment with BoW, TF-IDF, GloVe, and Word2Vec features are shown in Fig. 4. SVM is significant with all features and achieved the best score with BoW, TF-IDF, and Word2Vec. This significant performance of SVM is because of its linear architecture and binary classification problem. SVM is more significant on linear data for binary classification with its linear kernel as shown in this study.

Results on TextBlob annotated dataset

The contradictions in the users’ expressed sentiments in the reviews and assigned labels can influence the sentiment classification accuracy of the models. To resolve this issue, TextBlob annotated data are used for the performance evaluation of the models. Results suggest that the performance of machine learning models is better when used with TextBlob labels than the original labels.

Results using Word2Vec features

Table 21 shows the performance of machine learning models with Word2Vec features and TextBlob sentiments. SVM achieves significantly better accuracy with Word2Vec features as compared to GloVe features. It gives a 0.88 accuracy score which is more than GloVe features but lower than BoW and TF-IDF features. RF and GBC achieve 0.79 and 0.70 accuracy scores, respectively. The performance of DT is degraded when used with Word2Vec features.

Table 21:

Performance evaluation of classifiers using Word2Vec features on the TextBlob annotated dataset.

Model	Accuracy	Precision			Recall			F1 Score
		Pos.	Neg.	W avg.	Pos.	Neg.	W avg.	Pos.	Neg.	W avg.
DT	0.69	0.78	0.87	0.83	0.99	0.24	0.62	0.87	0.48	0.62
RF	0.79	0.78	0.87	0.83	0.99	0.24	0.62	0.87	0.38	0.63
GBC	0.70	0.80	0.44	0.62	0.80	0.44	0.62	0.80	0.44	0.62
SVM	0.88	0.90	0.83	0.87	0.95	0.71	0.83	0.92	0.77	0.84

DOI: 10.7717/peerj-cs.914/table-21

The comparison between machine learning model results on TextBlob sentiment dataset with BoW, TF-IDF, GloVe, and Word2Vec features is given in Fig. 5. SVM obtains better results with TextBlob sentiments using BoW and TF-IDF features as compared to GloVe and Word2Vec features. Similarly, the performance of other models such as RF, GBC, and DT has also been improved with the TextBlob sentiments. For the given dataset, the performance of models is good using TextBlob but these results can not be generalized for every other dataset. It is possible that a few original labels are not correct and using the TextBlob label can show better performance.

The performance of machine learning models is good when used with TF-IDF features extracted from the original dataset and SVM outperforms with a significant 0.89 accuracy score. TF-IDF generates a weighted feature set as compared to BoW, GloVe, and Word2Vec features which helps to improve the accuracy of learning models. On the other hand, the accuracy of DT is reduced by 1% from 72% to 71% because DT is a rule-based model that performs well on simple term frequency as compared to weighted features. Weighted features introduce complexity in the DT learning process. SVM performs well because TF-IDF provides a linear feature set with the binary class which is more suitable for SVM that performs better being the linear model. The performance of machine learning models is improved with TextBlob data annotation. Machine learning models perform well with TF-IDF and BoW features and SVM obtains the highest accuracy of 0.92 accuracy score using TextBlob labels.

Performance of deep learning models

To compare the performance of the proposed approach with the latest deep learning approach, experiments have been performed using several deep learning models. For this purpose, state-of-the-art deep learning models are used such as LSTM, CNN-LSTM, and Gated Recurrent Unit (GRU). The architecture of used deep learning models is provided in Fig. 6. These models are used with the TextBlob annotated dataset owing to the superior results on the dataset from machine learning models.

The performance of deep learning models is also good similar to machine learning models, as shown in Table 22. CNN-LSTM achieves a 0.90 accuracy score while GRU has a 0.86 accuracy. The results of deep learning models are shown in Table 6. Overall, the performance of deep learning models is slightly lower than the machine learning models. Regarding the machine learning model, SVM gives the highest accuracy of 0.92 while the deep learning model CNN-LSTM achieves a 0.90 accuracy. The significant performance of machine learning models is because of handcrafted TF-IDF weighted features.

Performance analysis with state-of-the-art approaches

Performance analysis has been carried out to analyze the performance of the proposed approach with other state-of-the-art approaches that utilize the IMDb movie reviews analysis. Comparison results are provided in Table 23. Results indicate that the proposed methodology can achieve competitive results to that of state-of-the-art approaches. The use of SVM with TF-IDF and BoW using the TextBlob technique provides an accuracy of 92% which is better than the state-of-the-art approaches.

Statistical T-test

The T-test is performed in this study to show the statistical significance of the proposed approach (Fatima et al., 2021). The T-test is applied to SVM results with the proposed approach and original dataset. The output from the T-test favors either null hypothesis or alternative hypothesis.

• Accept Null Hypothesis: This means that the compared results are statistically equal.

• Reject Null Hypothesis: This means that the compared results are not statistically equal.

The output values of T-test in terms of T-statistic and critical Value are shown in Table 24. T-test infers that the null hypothesis can be rejected in favor of the alternative hypothesis because the T-statistic value is less than the critical value indicating that the compared values are statistically different from each other.