An experimental study on the performance of collaborative filtering based on user reviews for large-scale datasets

State of the art

RSs have been used on several platforms, such as e-commerce and social networks. RSs use three main approaches to create recommendations: content-based, collaboration-based, and hybrid. RSs typically focus on two issues: predicting ratings and recommending items (Ricci, Rokach & Shapira, 2015; Fayyaz et al., 2020). This study focuses on rating predictions for the CF approach, the most popular approach used in RSs (Yang et al., 2016; Nguyen & Amer, 2019).

Collaborative filtering

CF generates a recommendation for a user based on the similarities among users who had similar preferences/interests in the past. This approach assumes that individuals with similar preferences in the past are likely to have similar preferences in the future (Aciar et al., 2007). CF identifies new user-item associations by determining user relationships and the interdependencies between items (Yang et al., 2016). Generally, a CF model consists of a set U of users and their preferences over a set of item I. Then, a utility function R measures the suitability of recommending item i ∈ I to user u ∈ U. It is defined as R: U ×I →R₀, where r ∈ R₀ is either a real number or a positive integer within a specific range (Adomavicius, Manouselis & Kwon, 2011).

CF can be grouped into two categories: memory-based and model-based (Chen, Chen & Wang, 2015). Memory-based CF can be motivated by the observation that users usually prefer recommendations from like-minded consumers. These methods apply nearest-neighbour-like algorithms to predict a user’s ratings based on the ratings given by like-minded users. These algorithms can be classified into user- and item-based methods (Adomavicius & Tuzhilin, 2005). The former identifies a set of neighbours for a target user and then recommends a set of items based on the neighbours’ interests. In contrast, the latter recommends items similar to (share features with) items that the user has previously purchased, viewed, or liked. The three most frequently used metrics/measures to identify user/item similarities are the Pearson correlation, Euclidean distance and Cosine-based similarity (Liu, Mehandjiev & Xu, 2011; Amer, Abdalla & Nguyen, 2021). On the other hand, the model-based CF predicts users’ ratings of unseen items by developing a descriptive model of user preferences and then using it for predicting ratings. Many of these methods are inspired by machine learning algorithms.

Generally, the performance of the CF approach depends on the availability of sufficient user ratings (Aciar et al., 2007). Such a dependency, however, may cause CF to suffer from sparsity, cold start, scalability, and rating bias problems (AL-Ghuribi & Noah, 2019; García-Cumbreras, Montejo-Ráez & Díaz-Galiano, 2013). The sparsity issue has attracted much attention, and various approaches have been proposed to solve it. One of the approaches is to extract valuable information from user reviews using sentiment analysis techniques and integrate them into CF (Chen, Chen & Wang, 2015). The following section provides a brief overview of sentiment analysis.

Sentiment analysis

Sentiment analysis or opinion mining is the computational study of people’s opinions, sentiments, emotions, appraisals, and attitudes towards entities such as products, services, organizations, individuals, issues, events, topics, and attributes (Liu, 2012). Approaches to sentiment analysis may involve various fields, such as natural language processing (NLP), information retrieval and machine learning. Sentiment analysis can be applied at three primary levels: the document level, the sentence level, and the aspect level, and the approaches can be roughly divided into two categories: machine learning and lexicon-based approaches. Both aim to determine or classify the user sentiments associated with each entity in the given text. Generally, most document-level and sentence-level sentiment classification methods rely on machine-learning approaches. In contrast, aspect-based sentiment classification methods are mostly lexicon-based approaches. The machine learning approach mainly involves extracting and selecting a proper set of features to detect opinions (Liu, 2012). Lexicon-based approaches primarily depend on a sentiment lexicon, which is a collection of precompiled known sentiment words (Serrano-Guerrero et al., 2015). Lexicon-based approaches can be either dictionary-based or corpus-based (Darwich, Noah & Omar, 2020).

Incorporating sentiment analysis into collaborative filtering algorithms

CF approaches rely on ratings and cannot function effectively without enough ratings. However, user reviews can serve as an alternative source in such cases by allowing sentiment analysis techniques to generate the overall rating. Leung, Chan & Chung (2006) were among the first to point out the possible benefits of combining sentiment analysis with CF approaches to increase accuracy by inferring ratings from user reviews when explicit ratings are unavailable. Following Leung, Chan & Chung (2006), Aciar et al. (2007) were the first study that attempted to use user reviews to build RSs by developing an ontology to transform review content into a standardized form that could be used to provide recommendations. More studies have proposed ways to integrate user reviews into CF to provide better recommendations with different combinations of other elements or enhanced with other models.

García-Cumbreras, Montejo-Ráez & Díaz-Galiano (2013) proposed an approach that classifies users into two classes (Pessimist and Optimist) according to the average polarity of the user reviews. The classes are included as a new attribute to the CF algorithm. Evaluation using a newly created corpus from the Internet Movie Database (IMDb) shows the enhanced the CF performance.

Zhang et al. (2013) proposed an algorithm to calculate an overall rating (called the “virtual rating”) from user reviews by aggregating the sentiment words with the reviews’ emoticons. They proposed a self-supervised sentiment classification approach involving both lexicon-based and corpus-based models that could determine the overall sentiment polarity of reviews containing textual words and emoticons.

Wang & Wang (2015) proposed an approach that mapped extracted opinions from online reviews into preferences in a way that CF-based RSs could understand. The experimental results for their proposed CF approach indicated that incorporating sentiment analysis into CF improves the accuracy of product recommendations.

Pappas & Popescu-Belis (2016) consider the user reviews to address the problem of one-class CF problem, which only handles explicit positive feedback. Extracted sentiment information from reviews is mapped as sentiment scores to user ratings using an integrated nearest neighbour model. Their proposed approach demonstrated that users’ unary feedback (likes or favourites) and the sentiments expressed in user reviews are inextricably linked.

Dubey et al. (2018) proposed an enhanced item-based CF approach using sentiment analysis. They built a dictionary, calculated the sentiment scores by the positive reviews’ probability, and filtered out items with negative reviews. The proposed approach improved the quality of predictions compared to conventional RSs. The limitation of this work is that the authors claim that their method of calculating the sentiment score based on probabilities is inaccurate because many reviews are classified incorrectly.

Ghasemi & Momtazi (2021) proposed a review-based model to boost CF by measuring user similarities based on seven different methods, including two that are lexicon-based, two that use neural representations of words, and three that are based on neural representations of text. Among these, the model based on a long short-term memory (LSTM) network achieved the best results.

Even though previous studies have found that sentiment analysis benefits the performance of the CF approach, the existing studies are still inadequate. Most of these studies calculate the implicit ratings using the conventional method that aggregates sentiment scores in a review or employing a classification method based on most positive or negative words. Both methods ignore the degree of sentiment words during calculating the implicit rating, which is not the case in this study. Furthermore, most recent studies that attempt to determine implicit ratings rely only on the sentiment words appearing in user reviews. In contrast, our study calculates the implicit rating using aspect–sentiment word pairs in addition to the method of aggregating sentiment words considering their degrees.

Other approaches that use user reviews go even further by proposing sentiment-based models that integrate contextual information to enhance the CF (Osman et al., 2021) or generate user profiles to identify user preferences (Cheng et al., 2019; Bauman, Liu & Tuzhilin, 2017; Wang, Wang & Xu, 2018). Moreover, others like (Da’u et al., 2020; Ray, Garain & Sarkar, 2021; AL-Ghuribi et al., 2023; Cheng et al., 2018) focused on integrating aspect/feature elements of reviews into CF as multiple criteria to improve the CF performance. As can be seen from the current studies, little emphasis has been placed on using user reviews to determine the implicit rating and integrate it into the CF approach. This motivated us to use user reviews to calculate implicit ratings using different methods that are not previously indicated in the available studies.

Some researchers also claim that reviews are insufficient for acquiring user-personalized information and that information stored in the user-item rating matrix is needed because different users may write identical reviews but assign different ratings to the same item (Wang et al., 2019). Therefore, the present study proposes two rating variants that combine the explicit ratings from the rating matrix with different implicit ratings derived from user reviews to determine whether user reviews are sufficient for producing personalized recommendations or require explicit ratings. Finally, the present study focuses on a large-scale dataset, which is not the case in many previous studies, and applying those methodologies from past studies to a large-scale dataset could result in imprecise ratings and negatively influence rating predictions (Hasanzadeh, Fakhrahmad & Taheri, 2022).

To summarise, we proposed four variants of implicit rating in this study. The four variants of implicit ratings were integrated into various CF rating prediction algorithms and compared against the overall (explicit) ratings. The aims are twofold. The first is to assess the implicit ratings’ effectiveness in enhancing recommendation algorithms’ prediction accuracy. The second is to determine the influence of implicit ratings on the performance of prediction accuracy when combined with explicit ratings. The ideas behind these aims are due to the distinctions between the overall and user reviews-derived ratings. Overall ratings are relatively objective, focusing on general preference or satisfaction. They do not capture the reasons behind the rating, the specific aspects that influenced the rating, or the contextual factors that shaped the user’s evaluation. User reviews, on the hand, capture the subjective nature of opinions. Users can express their preferences, perspectives, and subjective experiences in the review text. They provide context-specific insights and can highlight the factors that influence the user’s evaluation, such as specific features, quality, customer service, or price. Thus, the question of whether combining general preferences with the subjective nature of opinions can produce better prediction accuracy will be addressed by the experiment conducted in this study.

Method

This study aims to examine the effects of implicit ratings generated from user reviews compared to explicit ratings regarding the accuracy of CF rating prediction algorithms. We consider two variants of implicit ratings: ratings based on extracted sentiment word scores and ratings based on extracted aspect–sentiment word pairs. Apart from that, we also investigate the possibility of proposing new ratings by combining both implicit and explicit ratings. As such, the general methodology of the experimental study involved three tasks, as shown in Fig. 1.

• Task 1: Calculate two variants of the implicit rating extracted from user reviews.

• Task 2: Produce two additional variants for the overall rating:

  • The first variant combines the explicit and implicit ratings based on extracted sentiment word scores.

  • The second variant combines the explicit and implicit ratings based on aspect–sentiment word pair scores.

• Task 3: Integrate the generated ratings into various CF rating prediction algorithms and evaluate their accuracy compared to the explicit ratings.

The following subsections describe these tasks in detail.

Task 1: Generating implicit rating

This task aims to compute an implicit rating, which is the total review sentiment score, determined by analyzing users’ reviews that reflect their opinions of the consumed item. There are two methods for calculating implicit rating, based on either sentiment words or aspect–sentiment word pairs. Descriptions of the two methods follow.

Implicit rating using sentiment words

This method involves extracting all sentiment words mentioned in a review and calculating their scores based on a specific lexicon. The overall implicit rating is the sum of the scores of all extracted sentiment words (SW) mentioned in review i: (1)${\displaystyle ImplicitRating\text{\_}S{W}_i=\sum _{j=1}^{N}Score\left(S{W}_j\right)}$

where N is the number of sentiment words in review i and Score(SW_j) is the sentiment score for the sentiment word j mentioned in the review.

The selection of sentiment words and the assignment of their scores are based on our earlier work presented in AL-Ghuribi, Noah & Tiun (2020a). In this work, the sentiment words were selected from four parts of speech: noun, verb, adjective, and adverb because these parts of speech contain sentiment that can considerably influence the sentiment analysis process. The work (AL-Ghuribi, Noah & Tiun, 2020a) also provided a domain-based lexicon containing numerous sentiment words with their scores, which are used for assigning the score of Score (SW_j).

Implicit rating using aspects

The second method for calculating implicit ratings relies on aspect–sentiment word pairs. It is based on the hypothesis that the implicit rating is the weighted sum of the user’s opinions (i.e., ratings) about multiple aspects. Many experiments have shown this hypothesis to be reliable (Ngoc, Thu & Nguyen, 2019; Yu et al., 2011; Wang, Lu & Zhai, 2010; Al-Ghuribi, Noah & Tiun, 2020b).

To achieve this, the aspects are first extracted from reviews using the Semantically Enhanced Aspect Extraction (SEAE) method proposed in Al-Ghuribi, Noah & Tiun (2020b) and the blocking technique mentioned in Thabit & AL-Ghuribi (2013). Then each aspect’s weight is determined using the Modified Term Frequency–Inverse Document Frequency (TF-IDF) proposed by Zhu, Wang & Zou (2016). Finally, the domain-specific lexicon proposed in AL-Ghuribi, Noah & Tiun (2020a) is used to determine the scores for the sentiment words corresponding to each aspect. The implicit rating produced by this method is the weighted sum of the ratings of sentiment words belonging to each aspect mentioned in a review, calculated as follows: (2)${\displaystyle ImplicitRating\text{\_}Aspec{t}_i=\sum _{j=1}^k{R}_j{W}_j}$

where k is the number of aspects in review i, R_j and W_j refer to the sentiment word’s rating and weight for aspect j in review i.

We provide an example from the Amazon movie dataset to further explain the difference between the two generated implicit ratings. Assume that a user gives the following review:

“This is a charming version of the classic Dickens tale. Henry Winkler makes a good showing as the “Scrooge” character. The casting is excellent and the music old but very relevant.”

According to the first method based on extracted sentiment word scores, the implicit rating is 1.1075, as shown in Table 1.

The second method, which is based on aspect-sentiment word pairs, yields an implicit rating of 0.1202. This rating is obtained as shown in Table 2, where each aspect’s weight is assigned based on the modified TF-IDF proposed in Zhu, Wang & Zou (2016), and the sentiment scores are assigned based on the domain-specific lexicon proposed in AL-Ghuribi, Noah & Tiun (2020a).

Task 2: Produce two variants for the overall rating

This task generates two new variants for the overall rating by combining the implicit rating produced in Task 1 with the explicit rating, O. These ratings are calculated as the average of the explicit and the implicit ratings. Thus, one of the new variants is the average of the explicit and implicit ratings using sentiment words for the given review, as given in Eq. (3). The other is the average rating of the explicit and the implicit ratings using aspect–sentiment word pairs for the given review, as shown in Eq. (4). (3)${\displaystyle AVG\text{\_}ImpS{W}_i=\frac{\sum _{j=1}^{N}Score\left(S{W}_j\right)+O_i}{2}}$ (4)${\displaystyle AVG\text{\_}ImpAspec{t}_i=\frac{\sum _{j=1}^k{R}_j{W}_j+O_i}{2}}$

Task 3: Evaluation of the proposed ratings

This study aims to improve CF performance in terms of rating prediction when ratings are derived from user reviews. In the first two tasks, four different ratings are generated. This task evaluates the effectiveness of these ratings in improving the performance of CF rating prediction algorithms by integrating them independently into various CF rating prediction algorithms. This study uses the rating prediction algorithms available in the Surprise library (Hug, 2020b). This library offers a variety of benchmark rating prediction algorithms dedicated to RSs. These algorithms range in complexity from the most basic to the most advanced. Table 3 lists all the algorithms used in this study.

As can be seen, there are five categories for rating prediction algorithms, starting with the most fundamental, like BaselineOnly and NormalPredictor. Follows by the k nearest neighbour (kNN) algorithms, which are mainly used for regression and classification (Abdalla & Amer, 2021; Noaman & Al-Ghuribi, 2012; Al-Ghuribi & Alshomrani, 2013), and have evolved as one of the most popular fundamental algorithms for CF recommendation systems. There are four different kNN algorithms: the basic algorithm (kNNBasic), the algorithm that considers the mean ratings of each user (kNNWithMean), the algorithm that considers the normalization of each user’s z-score (kNNWithZScore), and the basic algorithm that considers a baseline rating (kNNBaseline).

Three algorithms fall under the category of “matrix factorization”: Singular Value Decomposition (SVD), another version of SVD that takes into account implicit ratings (SVDpp), and an algorithm based on Non-negative Matrix Factorization (NMF). The last two algorithms are SlopeOne, a simple implementation of the SlopeOne algorithm, and Co-clustering, based on the co-clustering technique. It is beyond the scope of this article to describe the details of all these algorithms; further descriptions are available in Hug (2020a).

Simply put, in this task, the proposed ratings are independently integrated into the 11 rating prediction algorithms listed in Table 3 to evaluate their impacts on improving the CF performance.

Evaluation of the results is based on three predictive accuracy metrics: mean absolute error (MAE), mean square error (MSE), and root mean square error (RMSE) metrics. A lower value of these metrics indicates a higher CF performance because they calculate the difference between predicted and actual ratings (Al-Ghuribi & Noah, 2021). The equations of the three metrics are as follows. (5)${\displaystyle \mathrm{MAE}=\frac{\sum _{i=1}^S\left(p_i-r_i\right)}{S}}$ (6)${\displaystyle \mathrm{MSE}=\frac{\sum _{i=1}^S{\left(p_i-r_i\right)}^2}{S}}$ (7)${\displaystyle \text{RMSE}=\sqrt{\frac{\sum _{i=1}^S{\left(p_i-r_i\right)}^2}{S}}}$

where S is the size of the test set, p_i is the predicted rating calculated by the CF approach, and r_i is the actual rating given by the user.

This study is the first to demonstrate various methods for generating implicit ratings on large-scale datasets. As a result, the explicit rating provided in the dataset is used as the baseline rating. Most of the available studies develop various CF models using explicit ratings. Specifically, the performance of each rating prediction algorithm mentioned in Table 3 using the proposed ratings is evaluated against the performance of each algorithm using the baseline rating (i.e., the explicit rating). The purpose of using the explicit ratings is to see whether the proposed ratings will outperform the explicit rating in enhancing the CF performance.

The Dataset

The proposed review-based CF approaches are evaluated using the movie domain of the Amazon dataset (http://jmcauley.ucsd.edu/data/amazon/links.html) (McAuley & Leskovec, 2013) and the restaurant domain of the Yelp dataset (https://www.yelp.com/dataset). Both datasets are considered as enormous datasets used to assess RSs (Seo et al., 2017). The movie and restaurant domains are the most widely used and popular in recent studies (Da’u et al., 2020; Al-Ghuribi & Noah, 2021; Dridi, Tamine & Slimani, 2022; Nawi, Noah & Zakaria, 2021). Figure 2 shows an example of the Amazon and Yelp datasets. Each review in both datasets is rated in the range of one and five, with one and two ratings denoting negative reviews, three denoting neutral reviews, and four and five ratings denoting positive reviews (Labille, Alfarhood & Gauch, 2016). Table 4 describes the datasets used in this study, and Fig. 3 shows the rating distribution for both datasets. As can be observed, over 800,000 movies in the Amazon dataset and over 400,000 restaurants in the Yelp dataset received five-star reviews.

The experiment setting

Three parameters must be identified before the three tasks of this study can be carried out. The parameters are as follows:

• λ–The minimum number of rated items (movies/restaurants) per user and the minimum number of ratings for each item (movie/restaurant).

• ξ–The rating scale is the standard rating scale to which all generated ratings are rescaled from their original rating scales.

• k–The optimal number of neighbours that will be used in all kNN rating prediction algorithms.

Different experiments are conducted to determine the optimal value for each parameter, with each experiment utilizing five-fold cross-validation. In each fold, the dataset is divided into training and testing (80% and 20%, respectively). The following subsections describe the conducted experiments.

The minimum number of rated items per user and ratings for each item (λ)

As indicated in Table 4, the Amazon dataset has 123,960 users, 1,697,471 reviews, and 50,052 movies, whereas the Yelp dataset has 401,867 users, 1,000,000 reviews, and 145,636 restaurants. The parameter λ is meant to choose users who have written enough reviews to ensure that the proposed approaches work effectively. It is also used to select movies/restaurants with a minimum number of reviews. For example, λ = 10 selects users who have rated at least ten movies/restaurants and selects movies/restaurants with at least ten users who have rated. For testing purposes, we chose five different values for this parameter: 10, 15, 20, 30, and 50. Table 5 shows the number of reviews, users, and items for each λ value for both datasets.

Table 4:

Details of the used datasets.

Attribute	Amazon	Yelp
Number of records	1,697,471	1,000,000
Number of unique product IDs	50,052	145,636
Number of unique users	123,960	401,867
Average number of ratings per user	13.69	2.49
Average number of ratings per product	33.91	6.87

DOI: 10.7717/peerjcs.1525/table-4

Table 5:

Dataset sizes based on different values of the parameter λ.

Value of λ	Number of records/reviews	Number of users	Number of movies
10	1,016,863	35,106	28,620
15	787,512	19,708	21,435
20	650,145	13,214	17,022
30	482,615	7,426	11,908
50	319,524	3,684	7,251
Amazon dataset

Value of λ	Number of records/reviews	Number of users	Number of restaurants
10	162,723	11,831	20,358
15	94,823	5,727	12,950
20	59,949	3,275	8,973
30	29,248	1,458	5,038
50	9,985	527	2,155
Yelp dataset

DOI: 10.7717/peerjcs.1525/table-5

Different experiments for the CF rating prediction process using the Surprise library (Hug, 2020b) are conducted to select the optimal value for the λ parameter. Each of the 11 rating prediction algorithms listed in Table 3 was applied to the five sub-datasets determined by the five possible values of λ, thus yielding 55 results for each dataset. All experiments used the overall rating (i.e., the explicit or baseline rating) provided in the dataset.

The MAE, MSE, and RMSE are used to evaluate the rating prediction accuracy of each algorithm with different λ values. Figure 4 illustrates the results of these metrics for the Amazon dataset.

The figure demonstrates that for all metrics, the dataset size has no impact on the order of prediction accuracies for the rating prediction algorithms (i.e., highest accuracy to the lowest accuracy). Mainly, for the five values of λ, the SVDpp algorithm provides the best results for the three metrics, followed by the BaselineOnly algorithm. On the other hand, the NormalPredictor algorithm gives the lowest accuracy, followed by the kNNBasic algorithm. The Yelp dataset shows a similar trend. Therefore, we chose λ = 20 for the Amazon dataset and λ = 10 for the Yelp dataset to obtain pertinent data (i.e., a fair number of records/reviews, users, and movies/restaurants).