Robustness of privacy-preserving collaborative recommenders against popularity bias problem

Privacy-preserving approaches for CF recommenders

Recent years have seen a significant increase in research into the privacy concerns raised by conventional CF systems. Many studies have concentrated on developing PPCF approaches to address these issues (Wei, Tian & Shen, 2018). These approaches are commonly designed using k-anonymity, obfuscation, differential privacy, cryptography, and perturbation-based techniques.

For distributed recommendation systems, cryptographic techniques are frequently utilized to protect the privacy of individuals (Badsha, Yi & Khalil, 2016; Li et al., 2017). For example, an unsynchronized secure multi-party computation protocol was introduced by Li et al. (2016). This protocol incrementally computes similarities between items without requiring individuals to be present when the calculation stages are performed. As another example, Shmueli & Tassa (2017) have offered a distributed recommendation strategy in which vendors communicate encrypted data with a mediator over secure protocols to calculate predictions or ranking items in generating recommendation lists. Also, some recent studies have used homomorphic encryption to encrypt the quality of service values to recommend personalized, high-quality web services based on user and location data without revealing confidential information (Badsha et al., 2018). Furthermore, Li et al. (2017) have proposed a successful approach for the group recommender systems that aim to safeguard users’ privacy while making recommendations. The process of computing similarities between items is considered a probabilistic inference issue in Zou & Fekri (2015), and a semi-distributed belief propagation network-based approach is introduced for item-based PPCF systems. This way, disclosure of user preferences is prevented by including a subset of individuals in the network at a specific time.

Another crucial method for protecting user privacy is anonymization, which aims to break the link between particular users and their rating profiles. For example, a micro aggregation-based PPCF solution is suggested by Casino et al. (2015) to ensure user privacy protection by using k-anonymity masks. Chen & Huang (2012) explore achieving k-anonymity for large-scale and sparse database kinds (e.g., RSs) and introduce a k-anonymity heuristic approach based on clustering for protecting privacy in data sharing. In addition, Wei, Tian & Shen (2018) propose a (p, l, a)-diversification technique for improving the performance of existing k-anonymity methods, where p denotes previous information on the rating profile of the users and (l, a) denotes user diversity to increase the degree of provided privacy. As the last example of the anonymization-wised method, Zhang, Lee & Choo (2018) apply this strategy for a scenario of privacy-aware smart cities and model privacy issues in cities as a PPCF issue.

Differential privacy is another prominent successful mechanism employed in CF-based approaches to quantify the loss of privacy that may occur when a user’s personal preferences are considered when generating a suggestion. Dwork et al. (2006) is the pioneering study propounding the differential privacy concept, and it is later used by McSherry & Mironov (2009) for the RSs domain by randomizing actual user preferences using a private covariance matrix. A more recent study (Guerraoui et al., 2015) offers a distance-based differential privacy solution to alter real profiles by exchanging components at specific distances within the profile. Shen & Jin (2014) propose an alternative method that combines noise calibration, ensures theoretical privacy and utility, and allows for adjustable instance-based noise injection into the preference profiles of the users. Finally, Hou et al. (2018) suggest a two-stage methodology for neighbourhood-based differential privacy recommendations and private neighbour selection in the context of medical recommendations.

In CF systems, data modification techniques like data obfuscation and randomized perturbation are the most used to protect privacy in the literature. More specifically, the data obfuscation technique tries to conceal private information while maintaining access to the information required for making recommendations. For example, Parameswaran & Blough (2007) present a permutation-based data obfuscation technique for central server-based CF applications. Also, a user-based PPCF recommender utilizing semi-honest parties to execute additional calculations is proposed by Badsha et al. (2017). For distributed recommendation contexts, Boutet et al. (2016) suggest employing coarse-grained obfuscated versions of user profiles to reveal private information. Also, Elmisery & Botvich (2017) extend traditional sole obfuscation level-based methods to identify more reliable target users by considering arbitrary obfuscation degrees.

Randomized perturbation techniques (RPTs), an alternative to previous approaches, can safeguard user privacy by falsifying actual user preferences before they are sent to the recommendation server (Polat & Du, 2003; Polat & Du, 2005a; Polat & Du, 2005b). For example, Bilge & Polat (2013) develop a scalable PPCF technique based on bisecting k-means clustering and randomized perturbations. In another study (Gong, 2011), the privacy of user-profiles spread across various repositories in recommendation processes is protected by combining RPTs with secure multi-party computation techniques. By utilizing various levels and ranges of random values, Polatidis et al. (2017) suggest enhancing the RPT-based PPCF process. In addition, Liu et al. (2017) develop a hybrid approach to privacy preservation that combines RPTs with differential privacy to offer more robust user preference protection than current RPT-based approaches. Finally, Yargic & Bilge (2019) apply RPTs to protect individuals’ privacy in the multi-criteria recommender systems domain.

Apart from the RSs domain, some prominent privacy-protection mechanisms also exist, especially for machine learning, particularly federated machine learning (Yin, Zhu & Hu, 2021). For example, Xu et al. (2019) have introduced HybridAlpha to protect against privacy risks in federated learning using secure multiparty computation (SMC) protocol based on functional encryption. Another recent study offers a hybrid technique combining SMC and differential privacy to cope with the trade-off between accuracy and small data in parties, including small amounts of data (Truex et al., 2019). Also, Kerkouche et al. (2021) have introduced a privacy protection mechanism for federated learning in the context of medical research and patient care, relying on differential privacy to improve bandwidth efficiency by enabling confidentiality. Finally, Lu et al. (2022) introduce a federated learning approach to detect malfunctions that may occur in wind turbines in an organization in the energy sector. To ensure inter-institutional data privacy, they have enriched their proposed approach with a biometric authentication technique to reduce privacy-preserving risks.

As it is known, RSs typically need rating data of individuals to produce appropriate referrals to them. Therefore, the size/dimension of data collections for prediction purposes directly affects the recommendation quality; the need for big data also brings two different problems. One is the transfer, analysis, and processing costs of such massive data. Second, it is not easy to ensure the confidentiality of big data and protect user privacy. Some recent studies, therefore, have also been carried out on using distributed environments to protect data privacy (Omar et al., 2019; Bosri et al., 2020; Lin, Tian & Liu, 2021). For example, Bosri et al. (2020) introduce a privacy-preserving recommendation system, Private-Rec, by integrating artificial intelligence strategies with blockchain technology. This system provides a secure environment for users by utilizing distributed features where users’ data is employed with their permissions. Another recent study introduces a blockchain-based recommender system platform that prioritizes privacy preservation (Omar et al., 2019). The decentralized nature of blockchain allows clients to control their data and consent to its use in the analysis. This platform incentivizes clients to share their information by offering rewards such as points or discounts, which can be used for relevant recommendations provided by the online company. Finally, Lin, Tian & Liu (2021) develop a blockchain-based recommendation system using local sensitive hashing and differential privacy techniques in the blockchain environment. Thus, the cost of prediction calculations decreased while the quality of the recommendation and the level of user privacy increased.

Popularity bias issue of CF recommenders

One of the most critical topics in RSs is how fairly the system acts for two essential stakeholders, i.e., users and item/service providers, without propagating any bias in provided referrals (Abdollahpouri & Burke, 2019; Yalcin, 2022a). However, recent studies reveal that recommendation algorithms mostly fail to handle items in their produced ranked lists fairly due to several reasons, such as their internal mechanisms or characteristics of data where they are trained (Chen et al., 2020).

One well-known example of such an issue is the popularity-bias problem, which leads to recommendation lists dominated mostly by popular items, even if they would not be extremely desired (Abdollahpouri et al., 2020; Yalcin & Bilge, 2022). This problem, unfortunately, causes significant decreases in the beyond-accuracy quality of the recommendations, such as diversity, catalogue coverage, or novelty, and worse weakens the robustness of the system against some malicious attacks. Therefore, there has been a rising interest in scrutinizing its adverse effects on recommendations and developing practical popularity-debiasing methods to achieve more appropriate recommendations (Abdollahpouri, Burke & Mobasher, 2019; Yalcin & Bilge, 2021; Boratto, Fenu & Marras, 2021).

The initial research on this problem has mainly discovered its effects on recommendations for different application domains such as movies, music, travelling, and online courses. For example, Jannach et al. (2015) consider several CF recommenders with different categories, such as matrix factorization- and neighbourhood-based and observe that they intrinsically feature a small number of popular items repeatedly in their recommendations. They also analyze how finetuning parameters of such CF algorithms affect their bias toward popular items. Boratto, Fenu & Marras (2019) perform a similar analysis for massive open online courses by considering the same set of CF recommenders to analyze how they are biased towards the popularity of courses and their categories. Another pioneering study (Abdollahpouri et al., 2020) analyses how the effects of such an issue differ for users and item suppliers.

The literature also considers how the recommenders’ popularity bias unfairly affects individuals with different interest levels in item popularity. For example, in movie recommendations, Abdollahpouri et al. (2019) observe that this problem is more harmful to users interested in popular items in their original profiles when compared to selective ones who mostly rate niche ones. Such an analysis is later re-performed for music recommendations with a more extensive set of CF recommenders (Kowald, Schedl & Lex, 2020). Also, Yalcin & Bilge (2022) classify individuals based on the essential characteristics of their rating behaviour and observe that highly-interacting, selective, and hard-to-predict users receive extremely unfair recommendations, especially in terms of their original propensities on item popularity. Finally, Yalcin & Bilge (2023) have analyzed the effects of popularity bias on users with different personality traits in the big-five-factor model and explored that users avoiding new experiences and individuals who are less extroverted are subjected to more unfair recommendations regarding item popularity.

Researchers have been exploring ways to mitigate the issue of popularity bias in recommender systems. For this purpose, various methods have been developed that can be broadly classified into three categories: pre-processing, in-processing, and post-processing, depending on how they reduce bias in recommendations (Boratto, Fenu & Marras, 2021). Pre-processing methods typically aim to balance the distribution of user preferences by altering data used to train recommendation algorithms. One example is an approach that divides items into head and tail categories based on their popularity and uses different data sets for recommendations for each group (Park & Tuzhilin, 2008). Another pre-processing method introduced by Jannach et al. (2015) constructs user-item pairs where popular items are filtered out to achieve tuples where observed items are less-popular. After that, recommendation algorithms are trained on such constructed tuples. Additionally, a probability distribution function designed considering item popularity can give underappreciated items more prominence in recommended lists (Chen et al., 2018).

Several attempts have also been made to tackle the popularity bias problem in recommender systems, particularly through in-processing methods. These methods modify conventional recommendation algorithms to consider both relevance and popularity simultaneously; therefore, they are commonly considered algorithm-specific ungeneralizable solutions. Some prominent examples of these approaches include: (i) a method that estimates individual preferences for disfavored items and utilizes this information to penalize popular items during the recommendation process (Kamishima et al., 2014); (ii) an optimized variant of the RankALS algorithm that balances intra-list diversity and accuracy (Abdollahpouri, Burke & Mobasher, 2017); (iii) a personalized recommendation framework that estimates familiarity between items based on their popularity and prunes the most preferred ones in order to achieve a more balanced common-neighbor similarity index (Hou, Pan & Liu, 2018); (iv) a debiasing approach that minimizes the correlation between user-item relevance and item popularity, which helps to treat items in the long-tail equally (Boratto, Fenu & Marras, 2021); (v) and a recommendation method based on variational autoencoders that penalizes scores given to items based on their historical popularity to increase diversity in the recommendation results (Borges & Stefanidis, 2021).

On the other hand, post-processing methods for this issue can be applied to the output of any recommendation algorithm and involve re-ranking or creating a new list of recommendations that follows a specific constraint. One approach, proposed by Abdollahpouri, Burke & Mobasher (2018), involves calculating final scores by weighting the predicted ratings inversely proportional to the popularity of items and then using these scores to re-rank the recommended list. Similarly, Yalcin & Bilge (2021) propose two methods for popularity-debiasing in recommendations, multiplicative and augmentative, specifically tailored for groups of users rather than individuals. Both are based on a re-ranking strategy of items but differ in how they penalize items based on their popularity. More specifically, the multiplicative approach attempts to reduce the average popularity of the recommended items by using the item weights as a factor when computing the ranking scores. On the other hand, the augmentative approach takes prediction scores as the primary influence and adds the item weights as a factor to the ranking scores to achieve a reasonable level of accuracy in the final recommendations. Another renowned approach, presented by Abdollahpouri, Burke & Mobasher (2019), is the xQuad re-ranking approach, which balances the trade-off between ranking accuracy and the coverage of long-tail items.

As can be followed by the presented two sections, there is a rising interest in the RSs literature to protect individuals’ privacy and counteract the popularity bias of recommendation algorithms. However, to the best of our knowledge, there is no research analyzing how the bias propagation of CF recommenders differentiates when they are empowered with such privacy-protection mechanisms. In addition, it is required to perform more analyses of whether popularity-debiasing methods can still show similar performances when traditional CF recommenders are decorated with such privacy-preserving strategies.

Datasets

In the experiments, we use three publicly available real-world benchmark datasets collected for different application domains, as in a very related recent study (Yalcin & Bilge, 2022); MovieLens-1M (https://grouplens.org/datasets/movielens/1m/) (ML) for the movies (Harper & Konstan, 2015), Douban Book (https://www.douban.com/) (DB) for the books (Shi et al., 2018), and Yelp (https://www.yelp.com/) for the local business reviews (Shi et al., 2018). User preferences in all utilized datasets are discrete and employ a five-star rating scale. Table 3 presents detailed information about the ML, DB, and Yelp datasets. Also, they have varying dimensions and sparsity ratios, as can be followed in Table 3, enabling us to analyze how such two important aspects of the datasets are related to potential unfairness issues for user personas with different privacy levels.

Utilized recommendation algorithms

It is very important to enrich the experiments with different parameters to analyze better the potential unfairness issues for individuals preferring varying privacy levels. The most convenient way to achieve it is to employ different algorithms in the recommendation generation phase. Indeed, this process can be described as generating recommendation lists using different algorithms after performing data perturbation and obfuscation operations to see how the privacy protection measures affect each algorithm’s bias towards popularity.

Therefore, ten different algorithms, two non-personalized and eight state-of-the-art approaches of four different CF families (i.e., matrix factorization-based, probabilistic, clustering-based, and neural networks-based), have been determined for recommendation purposes, as detailed in Table 4. Two non-personalized methods, i.e., ItemAverage (IA) and MostPopular (MP), are selected due to their wide usage in the literature to provide straightforward recommendations (Boratto, Fenu & Marras, 2019; Yalcin & Bilge, 2022). More specifically, the IA recommends items with the highest average rating, while the MP sorts items in descending order based on the number of their received ratings and then considers the most popular ones as recommendable. Therefore, by its nature, MP produces the same recommendation lists for each individual and denotes the maximum level of popularity bias that would be propagated. On the other hand, the remaining eight CF algorithms are selected as the best-performing ones from different families and applied via the Python-based library called Cornac (https://cornac.preferred.ai/) (Salah, Truong & Lauw, 2020). Also, for reproducibility purposes, we left hyperparameters of the algorithms set to their default values. Note that since the primary motivation of the present study is to evaluate whether recommendation algorithms propagate disparities among privacy level-based user personas, we do not provide detailed information about these algorithms’ internal mechanisms for clarity.

Evaluation metrics

Since the study aims to measure the effect of perturbation and obfuscation processes on popularity bias in different aspects, nine essential evaluation metrics are considered through the experiments. One of them, i.e., Average percentage of recommended items (APRI), is introduced by Yalcin (2022b) to measure the level of popularity of recommended items solely. The remaining metrics can be grouped under two main categories, i.e., accuracy and beyond-accuracy. More specifically, we employ the well-known normalized discounted cumulative gain (nDCG), precision, recall, and F1-score metrics for measuring accuracy performances, while average percentage of long-tail items (APLT), novelty, long-tail coverage (LTC), and Entropy for assessing beyond-accuracy recommendation quality. Here, although the selected beyond-accuracy metrics focus on measuring different aspects of the provided recommendations, such as popularity, diversity, coverage, novelty, and inequality, all of them indeed associate with the well-known fairness concept in RSs literature from the item perspective (Abdollahpouri & Burke, 2019). We describe these metrics below in detail, where i₁, i₂, …, i_N denotes the items in the top- N list for user u, i.e., N_u, demonstrated by a recommendation algorithm.

Average percentage of recommended items (APRI):

This metric calculates the average popularity of recommended items by the algorithms; therefore, higher APRI values for top- N lists indicate that the recommendation algorithm propagates a high-level of undesirable popularity bias (Yalcin, 2022b). More concretely, this metric first calculates the popularity ratio of each recommended item i, i.e., P_i, by considering the ratio of the total number of users who rate the corresponding item to the number of all users in the system. Then, it computes the average of the P values of the items in the Top- N list to achieve an APRI_u score for the corresponding user u, as formulated in Eq. (1). (1)${\displaystyle APR{I}_u=\frac{\sum _{i\in N_u}{P}_i}{N}}$

Precision, recall, and F1-score:

To measure the accuracy quality of produced top- N recommendations, we employ well-known precision, recall, and F1-score metrics (Yalcin & Bilge, 2022). More specifically, the precision score of a top- N list (i.e., P@N_u) for a user u is calculated as the percentage of the recommended N items suitable for the user. On the other hand, the recall score of the corresponding recommendation list (i.e., R@N_u) is computed as the ratio of the number of suggested suitable items to the number of all rated items in u’s profile. Here, when determining whether a recommended item is suitable, we set the threshold value as 3.5, as the literature has reached a consensus that 4 and 5 are the affirmative ratings in a 5-star rating scale (Bobadilla et al., 2013). Finally, F1-score of the considered top- N list (i.e., F1@N_u) is calculated as the harmonic mean of its P@N_u and R@N_u scores, as formulated in Eq. (2); thus, it provides a balance between such two metrics measuring accuracy level in different aspects. (2)${\displaystyle F1@N_u=2\times \frac{P@N_u\times R@N_u}{P@N_u+R@N_u}}$

Normalized discounted cumulative gain(nDCG):

Another important accuracy metric used in the experiments is the nDCG, which considers the actual ratings of the recommended items and their positions in the produced top- N list (Yalcin & Bilge, 2022). Supposing that r_u,i is u’s actual rating provided for item i, the ${DCG}_{N_u}^u$ and ${nDCG}_{N_u}^u$ values of the top- N recommendation list of u is calculated using Eqs. (3) and (4), respectively. (3)${\displaystyle {\text{DCG}}_{N_u}^u=r_{u,i_1}+\sum _{n=2}^{N_u}\frac{r_{u,i_n}}{lo{g}_2\left(n\right)}}$ (4)${\displaystyle {n\text{DCG}}_{N_u}^u=\frac{{\text{DCG}}_{N_u}^u}{{\text{IDCG}}_{N_u}^u}}$ where ${IDCG}_{N_u}^u$ value is the maximum possible gain for u and is obtained by ideally ordering N items.

Average percentage of long-tail items (APLT):

Another metric used in the experiments to evaluate the quality of ranking-based recommendations is the APLT, measuring what percentage of the recommended items are from the long-tail part of the popularity curve (Abdollahpouri, 2020). In other words, it demonstrates the ability of the recommendation algorithms to suggest niche items. When categorizing items as tail or head, the most commonly utilized approach is the Pareto Principle (Sanders, 1987), labelling items that receive 20% of the total ratings as head and the remaining ones in the catalogue as the tail items. Suppose that T is the set of assigned tail items via this principle, the APLT score of a top- N recommendation list produced for user u is calculated as given in Eq. (5). (5)${\displaystyle APL{T}_u=\frac{\left|\left\{i,i\in \left(T\cap N_u\right)\right\}\right|}{N}}$ Note that although APLT_u reflects the diversification level belonging to a particular user. However, the overall APLT score is an average of such diversification levels, and the higher APLT scores do not necessarily mean that the considered algorithm can provide high-quality referrals in terms of diversity (Abdollahpouri et al., 2021). Nevertheless, lower APLT scores indicate a general lack of diversity in recommendation lists. Therefore, we also utilize Entropy as an alternative metric to measure the diversification level of all recommendation lists, which is explained in detail in the following.

Entropy:

This metric mainly measures how often the recommendation algorithms suggest each item in the catalogue (Elahi et al., 2021). In other words, it reflects the inequality level across the frequency distribution of the suggested items. Therefore, higher entropy values for algorithms indicate more homogeneity in frequencies and diversity in recommendations, which is more desirable, especially for system administration, to enable fairer competition in the market. More concretely, the produced top- N list for each individual is merged by including the duplicated items. Suppose that such constructed union of each top- N list is represented with N_all and the set of all items in the catalogue {i₁, i₂, …, i_K} is denoted with K, then the entropy value of a recommendation algorithm is computed using the formula given in Eq. (6). (6)${\displaystyle Entropy=-\sum _{i\in K}\left(Pr\left(i\right)\right)lo{g}_2\left(Pr\left(i\right)\right)}$ where Pr(i) denotes the relative frequency of item i appearing in N_all set.

Novelty:

As a prominent beyond-accuracy metric, we also use the novelty in the experiments, which can be considered as the ability of the recommendation algorithms to present items that users did not taste before (Yalcin & Bilge, 2022). In other words, it measures what percentage of a top- N list produced for a user u has not been previously rated; thus, the capability of the algorithms to provide novel alternatives to users. Suppose that I_u is the set of items rated by user u, the novelty score of top- N recommendations provided for the corresponding user, i.e., Novelty_u, can be computed using Eq. (7). (7)${\displaystyle Novelt{y}_u=\frac{\left|\left\{i,i\in N_u\text{and}i\cap I_u=0̸\right\}\right|}{N}}$

Long-tail coverage (LTC):

The last employed metric in the experiments is the LTC measuring how much the provided recommendations cover the tail part of the item catalogue (Abdollahpouri, 2020). We first construct a non-recurring aggregation of top- N lists produced for each user, which we refer to as ℕ. Contrary to the entropy calculation, the duplicated items in the individual top- N lists are eliminated here. Then, we determine an intersection item set, i.e., I_ℕ∩T, by considering items that appear both in ℕ and T simultaneously, where the latter is the tail item set determined via Pareto Principle (Sanders, 1987). Finally, we calculate the ratio of the size of I_ℕ∩T to the size of T, as in Eq. ??. Therefore, higher LTC values for algorithms indicate better coverage of the unpopular, i.e., tail, items in the recommendations. (8)${\displaystyle LTC=\frac{\left|I_{\mathbb{N}\cap T}\right|}{\left|T\right|}}$

Note that higher values of described metrics except for APRI indicate less-biased and more qualified recommendations in terms of accuracy and beyond-accuracy perspectives.

Experimentation methodology

In the experiments, we consider four scenarios described in Table 1, where users differ based on their desired privacy-protection level. Note that all experiments detailed in the following are realized for such four scenarios separately.

For each user profile scenario, we generate top-10 recommendations for users by using one of the previously explained algorithms. In doing so, we apply the well-known leave-one-out experimentation strategy (Kearns & Ron, 1997). Accordingly, we use a particular user as test data and the remaining ones as train data and compute predictions for all items for the user by running the considered algorithm on the train set. This process is performed repeatedly for each user in the dataset. Then, we select top-10 items for each user as the recommendation list based on their computed predictions in descending order.

To comprehensively evaluate how users concerned with varying privacy levels are differently affected by the recommendations, we compute precision, recall, F1-score, nDCG, APRI, and novelty value for the top-10 recommendation of each user and then calculate their averages to achieve final results. On the other hand, when computing the LTC and entropy performance of the algorithms, we consider the aggregation of all top-10 recommendations of users, as these metrics are not user-centric and operate on the set of all recommendations as a whole. Note that when reporting the experimental findings in the following section, we consider average values of 100 trials realized for each user-profile scenario to avoid potential inconsistencies during the randomization process of the perturbation and obfuscation phases.

The benchmark treatment methods for popularity bias problem

One of the main motivations of this study is to analyze how traditional popularity-debiasing methods’ performance varies when applied to disguised user profiles. To this end, we consider three benchmark post-processing treatment methods for the popularity bias issue concerning the recommendations’ different quality aspects. All methods aim to re-rank the recommendations by penalizing popular items and utilize a regularization parameter (i.e., λ) that weights predictive accuracy and one important beyond-accuracy perspective, such as diversity or coverage.

Two utilized popularity-debiasing methods are variants of the Enhanced Re-ranking Procedure proposed by Yalcin & Bilge (2021): Augmentative (Aug) and Multiplicative (Mul). Both approaches re-sort the items in the recommendation lists based on the synthetic ranking scores, which are estimated for items by inversely weighting their produced predictions and popularity ratios. However, they follow different mechanisms in producing such scores, i.e., Mul harshly penalizes popular items by weighting prediction scores inversely with the popularity ratio of items in a multiplicative way, and Aug uses prediction scores as a driving force by utilizing item popularities as minor factors. Thus, due to its design, the former can considerably improve beyond-accuracy quality, which might significantly decrease accuracy performance. On the other hand, the latter gives more importance to predictive accuracy while combating popularity bias issues as much as possible.

The third utilized algorithm is the famous xQuad (xQd) algorithm (Abdollahpouri, Burke & Mobasher, 2019), which focuses on achieving final recommendation lists for users by dynamically selecting items based on their categories: (i) head refers to the popular items, and (ii) tail refers to the unpopular (i.e., niche) items. The algorithm first selects a candidate item set by only considering the prediction scores produced by a recommendation algorithm. Then, it dynamically constructs the final recommendation lists based on the ranking scores of candidate items. Such scores are calculated at each iteration by weighting their prediction scores. The weighting process considers both the original propensities of users on the category of the corresponding item (i.e., head or tail) and the cover ratio of the current recommendation list for these categories.