Experimental interpretation of adequate weight-metric combination for dynamic user-based collaborative filtering

A. The MovieLens

Considering the current RS applications, the basic practical structure of data commonly has a user × item matrix format. One of the frequently trained scientific datasets is MovieLens (Harper & Konstan, 2015) which has several releases based on size and additional content.

Considering Table 2, the main types of MovieLens can be reviewed depending on the rating size. For example, ML100K has 100,000 clicks. The MovieLens dataset is upgraded several times for the expanded types and for the versions of previous releases. For instance, the ML100K type has various releases, such as one that includes one to five ratings with only decimal values. The latest ML100K version consists of 0.5 steps between ratings, including half stars. However, this version is not recommended for shared research results because it is a developing dataset. Several previous studies focused on the tried-and-trusted original ML100K release, which is a pioneering collection and has considerably efficient runtime performance. Considering the scope of this study, we utilize this original release, which includes only full stars. Additionally, we encapsulated extensive experiments of ML1M to maintain full-star rating scaling parallelism with the ML100K. Therefore, we comparatively present the results related to the original ML100K and ML1M.

In this section, preliminary dataset analyses are presented. To validate the methods used in the following sections, residual checks on user-based statistical arguments and item-based independence analyses were performed as follows.

• (1) Checking residuals on user-based statistical arguments

The dynamicity effect of user-based statistical arguments (such as the mean and median) is discussed in this subsection. As static and dynamic approaches are the main focus, a visualization of their residual analysis on the utilization of the arguments is presented. Therefore, the rating history of each user was examined based on the statistical observations. Considering any user, it is observed that the statistical values of all the ratings change when an item is assumed as unrated. The IOI values that are individually excluded from the user vector are dynamically processed. The effect of each discarded rating was recorded as a residual over the dynamic mean or median. Thereafter, the static observation and dynamic approach were evaluated using the residual approach.

Figures 2 and 3 show the static and dynamic analyses based on the (A) mean and (B) median usage based on the ML100K and ML1M releases, respectively. The x- and y-axis show the user ID and unique rating values, respectively. Each red dot statically indicates the mean or median values of all user ratings, whereas the blue dots show the deviation of the unit ratings from the static value. It may be observed in the median analysis that the blue dots were aggregated, while the outliers in the datasets were suppressed, indicating the superiority of the median over the mean in the presence of outliers.

• (2) Item independency analyses

Considering the dynamic approach regarding real-time systems, excluding the IOI in the users’ statistical calculations depends on the item’s independence condition. Therefore, each particular item in the datasets was analyzed based on the independence. The leave-item-out approach emerges as a useful method because the items are independent of each other. Consequently, an item-based one-way analysis of variance (ANOVA) was performed. Each column (i.e., each item) was subjected to testing in the user × item matrix, validating their independencies. The ANOVA provides information about inter- and intra-group variations. By calculating the sum of squares (SS), degrees of freedom (df), and mean squared errors (MS), the F-test (the ratio of inter- and intra-group variability) is applied. Considering Tables 3 and 4, an analysis of the ML100K and ML1M releases is presented. The validity of item independence is proven by both the $\mathrm{F}\gg 1$ and probability (P) values, which are obtained from the F-distribution. The lower the P-values, the higher the chances of strong evidence against the null hypothesis. P-values $\ll 0.05$ (significance level) were obtained, indicating that the null hypothesis is rejected.

B. Similarity and prediction equations

The four touchstone similarity equations and prediction formula are considered in this section. Before the technical statements, an overview of the application perspective of the touchstone equations is provided.

$PCC$ is within the scope of several studies. Music RS is among the most common applications (Kuzelewska & Ducki, 2013). In particular, considering music genre recommendations, $PCC$ has attracted attention. Additionally, there are other applications. Mukaka explains the management of medical data based on the utilization of $PCC$ (Mukaka, 2012), as in other studies (Akoglu, 2018; Miot, 2018). Apart from these, book recommendations via $PCC$ (Kurmashov, Latuta & Nussipbekov, 2016; Sivaramakrishnan et al., 2018), e-commerce applications (Lee, Park & Park, 2008), and academic paper RS (Lee, Lee & Kim, 2013) are intriguing alternatives. Other $PCC$ examples can be found in Adiyansjah, Gunawan & Suhartono (2019), Cataltepe & Altinel (2009), Sigg (2009) and Shepherd & Sigg (2015). The most common application is movie-based RS (Dhawan, Singh & Maggu, 2015; Madadipouya, 2015; Sheugh & Alizadeh, 2015), which is the motivation of the present work. Movie genre correlations were calculated using $PCC$ by Kim et al. (2010). Hwang et al. (2016) also presented the details of the $PCC$, considering movie genre classification. Nonetheless, $PCC$ has some disadvantages, considering the linear averaging procedures. Tan & He (2017) indicated the underlying limitations of the $PCC$ in reinforcing the effect of correlation. Subsequently, they proposed the resonance similarity between users by parametrizing the median of the rating values. They constructed a physical analogy between user similarities regarding simple harmonic motion in a coordinate system (Tan & He, 2017). The mean of the rating vectors can be vulnerable to outliers and biases, as Garcin et al. (2009) emphasized the superiority of median-based rating aggregations over mean- and mode-based aggregations. Therefore, the use of $MRC$ was proposed to suppress outliers in user ratings in the context of RS. $COS$ is another frequently used method in RS owing to its simple calculation, and is encountered in movie-related applications (Singh et al., 2020; Wahyudi, Affandi & Hariadi, 2017), research paper recommendation (Philip, Shola & John, 2014; Ahmad & Afzal, 2020; Samad et al., 2019), cognitive similarity-based design (Nguyen et al., 2020), article suggestion system (Rajendra, Wang & Raj, 2014), and music RS (Aiolli, 2013). Furthermore, $JAC$ has been evaluated in several studies (Bag, Kumar & Tiwari, 2019; Sun et al., 2017; Meilian et al., 2014; Rana & Deeba, 2019). $JAC$ has an essential feature regarding binary rating analysis (Zahrotun, 2016) and is considered a measure that does not treat absolute ratings (AL-Bakri & Hashim, 2019). From the perspective of merits and demerits of the relevant touchtone equations, the following detailed information is introduced. In general, $PCC$ provides a concept of the presence, absence, and degree of correlation. It also provides feedback on positive and negative correlations. However, computationally $PCC$ is complex owing to the complex algebraic requirements in its formula. It is also ineffective against outlier values. One merit of $MRC$ emerges with its median usage, whereas $PCC$ is based on the assumption of only linear correlation and is not the appropriate option for homogeneous data. Although $MRC$ shows similar features to $PCC$, it is a more suitable option, especially for data with outliers. However, demerits of $PCC$ are also valid for $MRC$. Using angle information, $COS$ easily calculates the correlation between data, which are even quite far in terms of Euclidean distance. However, $COS$ does not provide a concept about magnitude. Saranya, Sudha Sadasivam & Chandralekha (2016) emphasized that $COS$ is ineffective in capturing similar users who rated quite few items. Conversely, $JAC$ has the merits of binary set processing. The utilization of $JAC$ is simple because the equation requires only the set operations, especially for the binary ratings. However, if a system has vectors of categorical or multiple-valued data, then $JAC$ requires a preprocessing step for the binarization (Supriya; Saranya, Sudha Sadasivam & Chandralekha, 2016).

Together with the proposed nIOI and SW modifications over similarity equations, different combinations are interpreted by inferring the underlying affinity. Subsequently, the $PCC$, $MRC$, $COS$, and $JAC$ similarities are stated technically. Owing to the various performance metrics presented in Subsection “Performance metrics”, adequate weight-metric combinations are to be determined.

• (1) Pearson correlation

Pearson correlation is an acclaimed measure adopted in many data-mining approaches that address the similarity of measurement data. Regarding the user-based CF, the $PCC$ is a tool used to define in-between user similarity by considering the item ratings. Pearson weighs all connected neighbors and calculates the degree of a linear relationship between two users. Thus, a weight for each correlated neighbor is derived, achieving a linear relationship by processing the deviation from the mean values, $\overline{r}$ (Pearson, 1894). Considering Eq. (1), the similarity formula between two users, $a$ and $u$, is indicated.

(1) $$w_{a,u}^{PCC}=\frac{{\displaystyle {\sum }_{i\in ({\mathrm{I}}_a\cap {\mathrm{I}}_u)}\left(\left(r_{a,i}-{\overline{r}}_a\right)\times \left(r_{u,i}-{\overline{r}}_u\right)\right)}}{{\displaystyle \sqrt{{\sum }_{i\in ({\mathrm{I}}_a\cap {\mathrm{I}}_u)}{\left(r_{a,i}-{\overline{r}}_a\right)}^2}\times \sqrt{{\sum }_{i\in ({\mathrm{I}}_a\cap {\mathrm{I}}_u)}{\left(r_{u,i}-{\overline{r}}_u\right)}^2}}}.$$

• (2) Median-based robust correlation

Median-based robust correlation is a method that replaces the linear mean procedures with the median operation (Shevlyakov, 1997; Shevlyakov & Smirnov, 2011; Pasman & Shevlyakov, 1987). The utilization of the averages may suffer from the skewness problem (Pearson, 1895; Sato, 1997). In addition, outliers can affect mean values. The $MRC$, which has the median of rating values instead of the averages similar to those in $PCC$, represents the suppression of outliers in the ratings of each user. Considering Eq. (2), the $MRC$ formula is as follows.

(2) $$w_{a,u}^{MRC}=\frac{{\displaystyle {\sum }_{i\in \left({\mathrm{I}}_a\cap {\mathrm{I}}_u\right)}\left(\left(r_{a,i}-{\tilde{r}}_a\right)\times \left(r_{u,i}-{\tilde{r}}_u\right)\right)}}{{\displaystyle \sqrt{{\sum }_{i\in \left({\mathrm{I}}_a\cap {\mathrm{I}}_u\right)}{\left(r_{a,i}-{\tilde{r}}_a\right)}^2}\times \sqrt{{\sum }_{i\in \left({\mathrm{I}}_a\cap {\mathrm{I}}_u\right)}{\left(r_{u,i}-{\tilde{r}}_u\right)}^2}}}.$$

Contrary to the mean values of user ratings, ${\overline{r}}_a$ and ${\overline{r}}_u$ in Eq. (1), the median values, ${\tilde{r}}_a$ and ${\tilde{r}}_u$, represent the midpoints of the sorted ratings. The formula is similar to $PCC$, and the median point of the user ratings is considered as a neutral mark.

• (3) Cosine similarity

Another similarity is based on the cosine function. By performing the Euclidean dot product, the cosine value between the two n-element vectors $\mathit{A}$ and $\mathit{B}$ can be determined. Thus, the similarity is based on $\mathit{A}.\mathit{B}=\Vert \mathit{A}\Vert \Vert \mathit{B}\Vert cos\mathrm{\Theta }$. Considering the user-based similarity calculation via $COS$, the similarity weight between $a$ and $u$ can be measured as in Eq. (3).

(3) $$w_{a,u}^{COS}={\displaystyle \frac{{\mathit{r}}_{\mathit{a}}.{\mathit{r}}_{\mathit{u}}}{\Vert {\mathit{r}}_{\mathit{a}}\Vert \Vert {\mathit{r}}_{\mathit{u}}\Vert }.}$$

Some other versions of the conventional $COS$ have also been developed, such as the adjusted (Gao, Wu & Jiang, 2011) and asymmetric cosine similarities (Aiolli, 2013).

• (4) Jaccard similarity

Jaccard similarity is a measure of common elements in two sets. The rating history of a user under test, ${\mathrm{I}}_a$, and corresponding neighbor history, ${\mathrm{I}}_u$, are calculated using Eq. (4). The $JAC$ considers the two sets by having the ratio of their intersection to the union. The range of this similarity coefficient is 0 $\le w_{a,u}\le$ 1, where zero indicates that there are no common elements, whereas one implies that all the elements in the two sets are fully joint.

(4) $$w_{a,u}^{JAC}={\displaystyle \frac{\left|{\mathrm{I}}_a\cap {\mathrm{I}}_u\right|}{\left|{\mathrm{I}}_a\cup {\mathrm{I}}_u\right|}.}$$

• (5) Prediction equation

After the similarity calculations for all the best neighbor nominees, the obtained weights are sorted by denoting $w_{a,u^{\ast }}$. Thereafter, considering the sorted weights and $BNC$ limits, the best neighbors are determined. The prediction phase must be completed to achieve the recommendation score. The rating prediction formula, which is known as the mean centering approach (Saric, Hadzikadic & Wilson, 2009; Zeybek & Kaleli, 2018; Sarwar et al., 2001; Wu et al., 2013; Singh et al., 2020), is given in Eq. (5).

(5) $$p_{a,i}={\overline{r}}_a+\frac{{\displaystyle {\sum }_{u^{\ast }=1}^{BNC}\left(\left(r_{u^{\ast },i}-{\overline{r}}_{u^{\ast }}\right)\times w_{a,u^{\ast }}\right)}}{{\displaystyle {\sum }_{u^{\ast }=1}^{BNC}{w}_{a,u^{\ast }}}}.$$

C. Modified equations

Considering the equations given in the previous section, we modified these formulas. As a result, the efficiency of RS was significantly improved under some circumstances.

The modifications were made based on two aims, including (i) to create a system model suitable for real-time applications and (ii) to boost the similarity weights. The former is related to the dynamicity, whereas the latter is related to the user-of-interest and its neighbor by considering their $CIC$ as a constant multiplier, thereby signified weights can be obtained.

Considering the first phenomenon, the already rated test item was discarded from the user-of-interest rating history to predict the actual rating. Thus, during the mean or median calculations in formulas such as $PCC$ and $MRC$, the item is excluded as expected in real-time systems. This case is also valid for other measurements, such as $COS$ and $JAC$, where the related item is removed from the vectors in progress. To indicate this phenomenon, we use the nIOI subscript by denoting $\hat{a}$ in the equations. As explained in the first section, the negligence of the nIOI in many other RS applications is thought to be due to runtime concerns in vast scientific tests.

The second phenomenon is relative weight scaling, known as SW. This gives priority to a neighbor with more common ratings for the items. After calculating the co-rated item count, the weights in similarity calculations are signified using $CIC=\left|{\mathrm{I}}_a\cap {\mathrm{I}}_u\right|$, a constant multiplier (Okyay & Aygün, 2020). There are other alternatives in the literature (Ghazanfar & Prugel-Bennett, 2010; Levinas, 2014; Zhang & Yuan, 2017; Gao et al., 2012; Bellogín, Castells & Cantador, 2014; Zhang et al., 2020; Okyay & Aygün, 2020). For instance, Bellogín, Castells & Cantador (2014) compared different user-user weighting schemes. That set in this study is known as user overlap, which calculates common item counts between user neighbors (Raeesi & Shajari, 2012), considering Herlocker (Herlocker, Konstan & Riedl, 2002) and McLaughlin’s significance weightings (McLaughlin & Herlocker, 2004) together with trustworthiness (Weng, Miao & Goh, 2006) and trust deviation (Hwang & Chen, 2007). However, they either include extra parameters or require complex computations. Raeesi & Shajari (2012) compared the SW strategies by underlining the user overlap, which demonstrated a higher efficiency, considering the error rates, although there were few arguments to process.

Regarding the modifications, each equation from the previous section is updated using the two abovementioned phenomena. Considering $PCC$, based on Eq. (1), Eq. (6) is obtained by excluding test-item bias. Subsequently, Eq. (7) is the signified version of Eq. (1) by applying only SW.

(6) $$w_{a,u}^{PC{C}_{nIOI}}=\frac{{\displaystyle {\sum }_{i\in \left({\mathrm{I}}_{\hat{a}}\cap {\mathrm{I}}_u\right)}\left(\left(r_{\hat{a},i}-{\overline{r}}_{\hat{a}}\right)\times \left(r_{u,i}-{\overline{r}}_u\right)\right)}}{{\displaystyle \sqrt{{\sum }_{i\in \left({\mathrm{I}}_{\hat{a}}\cap {\mathrm{I}}_u\right)}{\left(r_{\hat{a},i}-{\overline{r}}_{\hat{a}}\right)}^2}\times \sqrt{{\sum }_{i\in \left({\mathrm{I}}_{\hat{a}}\cap {\mathrm{I}}_u\right)}{\left(r_{u,i}-{\overline{r}}_u\right)}^2}}},$$

(7) $$w_{a,u}^{PC{C}^{sw}}=\left|{\mathrm{I}}_a\cap {\mathrm{I}}_u\right|\times w_{a,u}^{PCC}.$$

The same approach is followed for the MRC in the Eq. (8). The SW multiplication is expressed in Eq. (9).

(8) $$w_{a,u}^{MR{C}_{nIOI}}=\frac{{\displaystyle {\sum }_{i\in \left({\mathrm{I}}_{\hat{a}}\cap {\mathrm{I}}_u\right)}\left(\left(r_{\hat{a},i}-{\tilde{r}}_{\hat{a}}\right)\times \left(r_{u,i}-{\tilde{r}}_u\right)\right)}}{{\displaystyle \sqrt{{\sum }_{i\in \left({\mathrm{I}}_{\hat{a}}\cap {\mathrm{I}}_u\right)}{\left(r_{\hat{a},i}-{\tilde{r}}_{\hat{a}}\right)}^2}\times \sqrt{{\sum }_{i\in \left({\mathrm{I}}_{\hat{a}}\cap {\mathrm{I}}_u\right)}{\left(r_{u,i}-{\tilde{r}}_u\right)}^2}}},$$

(9) $$w_{a,u}^{MR{C}^{sw}}=\left|{\mathrm{I}}_a\cap {\mathrm{I}}_u\right|\times w_{a,u}^{MRC}.$$

Regarding the $COS$, Eq. (10) shows the vector operations of the ratings in which the test-item bias is discarded, and the SW approach is described in Eq. (11).

(10) $$w_{a,u}^{CO{S}_{nIOI}}=\frac{r_{\stackrel{⌢}{\mathit{a}}}\cdot r_u}{\Vert r_{\stackrel{⌢}{\mathit{a}}}\Vert \Vert r_u\Vert },$$

(11) $$w_{a,u}^{CO{S}^{sw}}=\left|{\mathrm{I}}_a\cap {\mathrm{I}}_u\right|\times w_{a,u}^{COS}.$$

Finally, $JAC$ with the modifications is shown in Eq. (12) and (13).

(12) $$w_{a,u}^{JA{C}_{nIOI}}=\frac{{\text{|I}}_{\widehat{a}}\cap {\text{I}}_u\text{|}}{{\text{I}}_{\widehat{a}}\cup {\text{I}}_u},$$

(13) $$w_{a,u}^{JA{C}^{sw}}=\left|{\mathrm{I}}_a\cap {\mathrm{I}}_u\right|\times w_{a,u}^{JAC}.$$

Although previous studies (Ghazanfar & Prugel-Bennett, 2010; Levinas, 2014; Zhang & Yuan, 2017; Gao et al., 2012; Bellogín, Castells & Cantador, 2014; Raeesi & Shajari, 2012; Herlocker, Konstan & Riedl, 2002; McLaughlin & Herlocker, 2004; Weng, Miao & Goh, 2006; Hwang & Chen, 2007) have presented a good understanding of SW using different perspectives, there is a lack of detailed performance analyses in the relevant literature. We contribute to the relative comparison of similarity equations enhanced with $SW$, including their corresponding performance.

The two phenomena nIOI and SW were independently measured. Subsequently, to monitor the hybrid effect of these approaches, both are utilized by obeying the generalized formula in Eq. (14).

(14) $$w_{a,u}^{SIMILARIT{Y}_{nIOI}^{SW}}=\left|{\mathrm{I}}_{\hat{a}}\cap {\mathrm{I}}_u\right|\times w_{a,u}^{SIMILARIT{Y}_{nIOI}}.$$

The modified rating prediction formula is given by Eq. (15). Considering the $nIOI$, ${\overline{r}}_a$ is updated compared with the original equation in Eq. (5).

(15) $$p_{a,i}={\overline{r}}_{\hat{a}}+\frac{{\displaystyle {\sum }_{u^{\ast }=1}^{BNC}\left(\left(r_{u^{\ast },i}-{\overline{r}}_{u^{\ast }}\right)\times w_{a,u^{\ast }}\right)}}{{\displaystyle {\sum }_{u^{\ast }=1}^{BNC}{w}_{a,u^{\ast }}}}.$$

D. Performance metrics

The final phase of the proposed RS design involves monitoring the running algorithm. Because the CF is an intersection of statistics and machine learning, conclusive information on its performance is necessary. Particularly, understanding the inter-relational achievement of similarity equations with implied modifications requires thorough performance monitoring through numerous metrics. Regarding this, we focus on two main groups: well-known metrics and preeminent metrics. The former includes the frequently practiced performance monitoring, whereas the latter is less known in the literature, but still prominent for RS.

• (1) Well-known metrics

The well-known metrics in Table 5 are applied to the framework to provide insight for further studies. The explanations of the listed metrics are briefly summarized as follows.

First, exact accuracy is a metric used to measure the exact matches of actual ratings ( $r_{a,i}$) and corresponding predictions ( $p_{a,i}$). The accuracy computation is considered for the predicted rating, controlling whether $p_{a,i}=r_{a,i}$ or $p_{a,i}\ne r_{a,i}$. Considering the frameworks that use $N$-scale ratings, exact accuracy can provide a precise observation. In addition, threshold accuracy has also been used following the binary decision of liked and disliked items. By denoting $r_{a,i}\in {\mathbb{N}}^{+}$ or $r_{a,i}\in {\mathbb{R}}^{+}$ where $argmax(r_{a,i})=N$ over $N$-scale ratings, $t\in {\mathbb{R}}^{+}$ as a threshold value should be set satisfying $t<N$. Thereafter, the rating value $r$ compared to the threshold value $t$ is evaluated to label liked or disliked items in a binary sense (Bag, Kumar & Tiwari, 2019).

Second, correctly predicted positive values are measured via sensitivity, which is also known as the recall or true positive rate. In addition, the measure of the actual positives is monitored by precision, namely the positive predictive value. Precision is referred to by Powers (2007) as the true positive accuracy, indicating the confidence score. From sensitivity and precision, the F1-measure is calculated using the harmonic mean. On the contrary, aside from the positive decisions made, negatives have also been considered. Thus, specificity (or inverse sensitivity) represents the proportion of real negative cases. Moreover, inverse precision (or negative predictive value), which is also known as the true negative value by Powers (2007), shows the predicted negative instances. The false discovery and omission rates were deduced complementarily from the maximum metric scores of the precision and inverse precision. Shani & Gunawardana (2011) assert that the false discovery rate can be an alternative control mechanism, which is the proportion of $FP$ to the actual positives. Similarly, the false omission rate is the ratio of $FN$ to all negatives (Mukhtar et al., 2018).

Sensitivity and specificity are attributed as the true positive and negative rates, respectively. Similarly, the fallout and miss rate represent the false positive and false negative rates, respectively. The irrelevant recommendation ratio is obtained via the fallout. The miss rate is the ratio of the items that are not recommended although they are relevant. A recent study performed fallout and miss rate by practicing a personalized nutrition recommendation study (Devi, Bhavithra & Saradha, 2020a). Similar to the F1-measure, the utilization of precision and recall by means of geometric mean also appears in the Fowlkes–Mallows index. Considering another recent study, Panda, Bhoi & Singh (2020) discussed how to increase the Fowlkes–Mallows index similar to the F1-measure. Balanced accuracy provides a better perspective for performance analyses, considering an imbalanced confusion matrix. Balanced accuracy is the arithmetic mean of sensitivity and specificity. To understand the algorithm efficiency, utilizing several metrics, such as balanced accuracy, results in considerable feedback.

The final metrics are the threat score and prevalence threshold. The former considers hits, misses, and false alarms in the confusion matrix (Hogan et al., 2010). The latter emphasizes a sharp change in the positive predictive value. The prevalence threshold with a more geometric interpretation of the performance measurement with a focus on positive and negative predictive values is provided in Balayla (2020), and it has been applied to test analyses of Covid-19 screening (Balayla et al., 2020).

The metrics constructed from the confusion matrix and the error metrics, such as mean absolute error (MAE), mean squared error (MSE), and root mean squared error (RMSE), are considered in this study. Li et al. (2014), in their privacy-preserving CF approach, measured performance using RMSE and MAE as in Nguyen et al. (2020). The RMSE has demonstrated its efficiency in measuring error performance. For instance, considering the Netflix Prize competition, it was used as a vital indicator of the implementation (Bell & Koren, 2007).

• (2) Preeminent metrics

Although previous studies have given priority to F1-measure, precision, recall, and error-based measures (Bag, Kumar & Tiwari, 2019; Li et al., 2014; Nguyen et al., 2020; Hong-Xia, 2019), some other performance metrics which are relatively less recognizable in RS also provide robust decisions. According to Chaaya et al. (2017), well-known metrics cause significant biases, and markedness, informedness, and Matthews correlation are noteworthy alternatives. Because these preeminent metrics have limited utilization in the literature, they have been considered with a priority in this study. They consist of confusion matrix primitives as shown in Table 6. The definitions of these metrics are briefly reviewed below.

• (i) Markedness

The proportion of correct predictions is measured by markedness. This metric is free from an unbalanced confusion matrix. The markedness scores in the range [−1, +1] and its associated formula is demonstrated in Eq. (16). Markedness can be a substitution of precision, which can be used as a tool that shows the status of the recommendation and “chance” (Schröder, Thiele & Lehner, 2011). To the best of our knowledge, markedness is one of the least considered metrics in the literature in the scope of RS science, although it supremely supplies information related to positive and negative predictive values. For instance, this phenomenon is known as DeltaP in the field of psychology, and Powers confirms that markedness is considered as a good predictor of human associative judgments (Powers, 2007; Shanks, 1995).

(16) $$Markedness=Precision+InversePrecision-1.$$

• (ii) Informedness

The second preeminent metric is informedness, which includes sensitivity and its inverse as shown in Eq. (17). Informedness scores in the same range [−1, +1], as in markedness (Schröder, Thiele & Lehner, 2011). This metric is also known as the Youden’s index because it differs from the accuracy, considering imbalanced events in the confusion matrix. The returned score defines a perfect prediction by +1, or indicates the opposite by −1 (Broadley et al., 2018). The efforts of informedness in RS science are limited although there are intriguing applications that practice this promising metric. Pilloni et al. (2017) performed informedness in e-health recommendations. Considering hotel recommendations, informedness has also been used to check the performance of the multi-criteria system. Ebadi & Krzyzak (2016) set two performance metrics as prediction- and decision-based, where informedness was considered in the scope of decision-based metrics. Regarding another research field, Marciano, Williamson & Adelman (2018) utilized informedness in the context of genetic applications. This was considered as the relative level of confidence (Marciano, Williamson & Adelman, 2018). In addition, Layher, Brosch & Neumann (2017) measured the performance of neuromorphic applications by assessing informedness.

(17) $$Informedness=Sensitivity+InverseSensitivity-1.$$

• (iii) Matthews correlation

The Matthews correlation is a promising observation of binary labeling. Considering Eq. (18), a wide implicit observation is obtained with a score in the range of [−1, +1]. The interpretation of this metric considers the three focus points: perfect prediction, random prediction, and total disagreement between the actual and predicted values. According to the score range, each corresponding focus point is indicated by +1, 0, and −1, respectively (Boughorbel, Jarray & El-Anbari, 2017).

(18) $$\begin{array}{r}MatthewsCorrelation=\sqrt{\begin{array}{c}PositivePredictiveValue\times TruePositiveRate\times \\ TrueNegativeRate\times NegativePredictiveValue\end{array}}\\ -\sqrt{\begin{array}{c}FalseDiscoveryRate\times FalseNegativeRate\times \\ FalsePositiveRate\times FalseOmissionRate\end{array}}\end{array}.$$

Matthews correlation is a combination of informedness and markedness. The former considers how informed the classifier’s decision with knowledge is compared to “chance” (Powers, 2007). Alternatively, informedness is paraphrased as a probability with respect to a real variable rather than the “chance” (Powers, 2013). Conversely, markedness carries information on how possibly the prediction variable will be marked by the true variable (Layher, Brosch & Neumann, 2017). Overall, the Matthews correlation, as a geometric mean of informedness and markedness, shows the correlation between the prediction and true values. The occurrence of this metric is rare in the literature; nonetheless, some intriguing studies have been conducted, such as those on diet recommendations (Devi, Bhavithra & Saradha, 2020b).

A. Analyses of the preeminent metrics

First, the preeminent metrics (such as informedness, markedness, and Matthews correlation) and the F1-measure are utilized to show the comparative performance plots of all the similarity equations in Table 7 using each individual metric. The ML100K and ML1M releases were analyzed separately. Considering the plots, the x- and y-axis represent the fine-tuned BNCs and related metric output, respectively.

The statistical approach is depicted in this study to set a dynamic environment which requires a more adaptive procedure. The results, based on hypothetical computations, can only determine the maximum achievable top-performance of the dynamicity concept. We prove that dynamicity deviates from the maximum reachable results. In the subsequent figures, the dashed lines represent the theoretical perspective by including the global-only statistics, causing a fallacy. Moreover, the solid lines represent dynamicity with the nIOI approach. In addition, lines with diamond marks illustrate the results free from the SW approach, whereas the SW adjustment can be monitored through the unmarked lines.

Considering Figs. 4 and 5, performance plots of ML100K and ML1M are provided for the preeminent metrics and F1-measure. Each row of subplots was compared to an equation-dependent perspective. Analyzing the ML100K, the similarity equation with only the SW modification achieves the best results for the PCC lines in black. Regarding the MRC (the lines with a green color), the same dominance for the SW methodology may be observed through all the metrics. Although SW does not boost the performance of the COS in all the metrics, plots with SW in JAC show similar performance compared to those without SW.

Comparatively, we present the same metric performance throughout the similarity measures in the context of ML1M as shown in Fig. 5. Including only the dynamic COS, all other similarities with SW increase the F1-measure performance. However, the performance in informedness diminishes compared to that in Fig. 4. The top- and least-performing lines in each similarity measure remain the same for markedness, Matthews correlation, and F1-measure as in the ML100K. Nonetheless, regarding informedness, the effect of the SW in the PCC and MRC interchanges the least-performing similarity equation. Furthermore, dynamicity resulted in the same expected outcomes in the ML1M analysis.

B. Hybrid monitoring considering the preeminent metrics

This subsection depicts the overall comparison of the compelled dynamicity with the applied weight significance. Considering Fig. 6A, for ML100K, PCC is notably in the leading position compared to other similarity measurements. The ranking for the rest of the similarity measures is difficult to generalize because the lines interchangeably depend on the BNC. Regarding markedness, MRC starts with better performance for fewer BNCs; however, the trend reverses for BNC > 27. In addition, measurements other than MRC performed better for BNC > 40. A similar behavior is valid for the informedness and Matthews correlation; nevertheless, each of them has a relatively greater BNC threshold. Approximately, BNC > 75 for informedness and BNC > 60 for Matthews correlation decayed the performance of the MRC. Considering the F1-measure, a relatively stable performance was obtained for the measurements when BNC > 10. Ranking the performance of the equations, the F1-measure can be considered as the most stable metric independent of the BNC for the ML100K. This makes it appropriate comparisons of similarity measure performances without further BNC considerations.

Regarding Fig. 6B, the same hybrid monitoring of nIOI and SW is presented for the ML1M. The top-performing lines of the abovementioned preeminent metrics were still obtained via the PCC. The COS, compared to the others, has a relatively poor performance for the informedness metric. The performance ranking of the similarity equations remains more stable as a function of the BNC compared to the ML100K. There are slight interchanges between MRC and JAC only in informedness. Nonetheless, considering the others, the relative performance is independent of the BNC. Contrary to the significant interchanges in the ML100K, performance metrics maintain their relative positions in the ML1M. This stability finding can be a general interpretation, and it can be concluded that the larger the dataset is, the more stable the performance.

C. Extensive analysis of other metrics for hybrid monitoring

The subsequent figures illustrate the extensive analysis of other metrics frequently evaluated in the literature. First, the ML100K plots with hybrid monitoring are presented in Fig. 7.

Regarding accuracy-based metrics, both the exact (sensitive rating prediction) and binary (liked or disliked labeling) performances were monitored. The PCC had a relatively higher accuracy margin of approximately 0.05 for both metrics. Another accuracy calculation is performed extensively in this study. Considering balanced accuracy, the PCC outperformed for all the BNCs, whereas the MRC diminished. Similarly, the error metrics were measured, considering both exact and binary performances. As expected, the binary prediction error rates were lower than the exact prediction error rates. Both are plotted using MAE, MSE, and RMSE. Considering the binary prediction error, the PCC achieved the lowest possible error rates. Nonetheless, regarding the exact accuracy metric, the top performance for low error rates was interchangeable based on the neighborhood. Approximately BNC > 40, BNC > 20, and BNC > 20 for MSE, MAE, and RMSE, respectively, yielded lower error rates with the JAC measure.

In addition, the Fowlkes–Mallows index shows that PCC and COS achieve outstanding performances, whereas in general, the MRC has poor performance. Considering the threat score, the hits of user dislikes were not included; however, the ratio of liking matches concerning the misses is checked. Similarity measurement rankings are relatively stable after BNC = 10 and PCC was an adequate measure, while MRC fell behind. This implies that, after the BNC value of 10, the ranking of the metric values from similarity equations relative to the y-axis remains stable.

This study also discusses different performance checks based on the interdisciplinary applications discussed previously. Considering the context of RS, we propose the application of a prevalence threshold that sources compound information, including several associated metrics inferred as $\left(\sqrt{sensitivity\times fallout}-fallout\right)/informedness$. Similar to the error metrics, the lesser the prevalence threshold value is, the more exactitude accomplished. Although higher values are obtained in the COS compared to the others, the PCC has proven its superiority with lower rates.

Some metrics, such as sensitivity and specificity, provide information on how likely the top-n items match the user’s taste or vice versa. Correctly identified positives (i.e., sensitivity) perform well for the lower BNCs, and the COS measure leads with a peak of approximately BNC = 9. Correctly identified negatives (i.e., specificity) are distinctively better as the neighbor count increases for COS and JAC, whereas PCC and MRC are relatively stable.

The last two rows of the subplots are complementary metric couples. The metrics of sensitivity and miss rate; specificity and fallout; precision and false discovery rate; inverse precision and false omission rate complete each other. Considering specificity, precision, and inverse precision, PCC performs adequately, which can be verified from fallout, false discovery rate, and false omission rate.

The ML1M evaluation is presented comparatively to the previous findings of the ML100K. Regarding Fig. 8, the analyses are illustrated for the same evaluated metrics. The trend in the exact accuracy was similar to that of the ML100K with greater scores. The PCC puts a margin, whereas the others perform closer to each other with slightly lower values compared to the PCC. In addition, the exact prediction error metrics generally result in a reduced numerical range. The most erroneous metric is the COS, which is valid for both exact and binary prediction error metrics. The error performance in the ML1M is relatively stable, and the PCC is still a less fragile metric for binary prediction. Considering binary analyses, similarity measures are homogenously ranked again with the dominance of the PCC.

Furthermore, the Fowlkes–Mallows index shows an increased range of scoring with respect to the ML100K findings. Regarding the threat score, it was observed that the MRC significantly improved compared to the others. Considering the prevalence threshold, COS had a higher margin than the others compared to the performances in the previous analysis. Moreover, although the COS has a good sensitivity observation, it performs worse in terms of specificity and precision, as demonstrated in the previous findings. This indicates that the COS suffers from a true negative rate and positive predictive value.

The monitoring of smooth sensitivity in the ML1M is an important feedback because it is a component of some compound metrics. On the contrary, an indicative finding from the comparison of both releases is the behavior of the specificity and fallout metric couples. Considering the ML100K, a relatively more stable distribution is monitored for increasing the BNC values, whereas in the ML1M, the behavior becomes unstable. Lastly, whereas JAC for precision increases in the ranking compared to the previous findings, the opposite is valid for the inverse precision.

D. Adequate weight-metric combination of the top-performing BNCs

Having represented all the plots, we summarize the test results using a tabular structure for a compact heat-map presentation. Considering Tables 10 and 11, the performance metrics for each similarity measurement are visualized in a colored format² . Considering the preliminarily explanations in the third section, the selected BNCs achieving the top performances are added to the tables, highlighting the main motivation of our study: the decision of adequate weight-metric combinations. Each metric is processed through column-wise coloring to make the comparison easier. Therefore, each coloring is evaluated within its own column. This indicates that the same color may correspond to different values in other columns; nevertheless, only a single column should be considered to interpret the coloring for any metric. The comparison of the similarity methods in the vertical direction is targeted, considering the neighborhoods. At the end of each heat-map table, the minimum and maximum values referenced in the coloring of the relevant column are shown. The tables demonstrate the comparison by addressing the different correlation equations over the outstanding neighborhoods; thereby, comparing approaches such as dynamicity and SW, considering each independent metric. The cells shaded in green indicate the effectiveness of the appropriate combination. We present the results using both the SW-induced dynamic equations and plain dynamicity; hence, the effect of weight boosting is monitored.