Enhancing neural collaborative filtering using hybrid feature selection for recommendation

Traditional recommender systems

Many recommender algorithms are based on collaborative filtering (Adomavicius & Tuzhilin, 2005), which depends on users’ past behavior to make predictions. Collaborative filtering-based recommendations are divided into latent factor methods (Koren, Bell & Volinsky, 2009) and neighborhood-based methods (Sarwar et al., 2001). Neighborhood-based approaches utilize ratings directly to evaluate new items for users. Such models are based on the similarities between a user to a user or item to an item. The similarity between two items is measured as the probability of users rating those items’ similarly, which is usually based on the Pearson correlation. On the other hand, the latent factor models users and items as vectors using the same latent space by reducing the number of hidden factors. Latent factors compare users and items directly, where a user’s rating of an item is predicted using the inner product between related latent vectors.

Singular value decomposition (Koren, Bell & Volinsky, 2009) reduces the number of user-item features to a product of two low-rank matrices. However, one of its drawbacks is the high cost of locating singular value decomposition. Another approach that enhanced the singular value decomposition, SVD++ (Koren, 2008), uses implicit and explicit feedback to provide a recommendation and demonstrate improved performance over many matrix factorization models.

Generally, traditional recommender algorithms use a linear kernel that does not better represent the user-item relationship.

Deep learning base recommender system

Deep learning has developed extensively in the past decade and has been implemented in various fields, such as computer vision, speech recognition, and natural language processing (Zhang, Yu & He, 2018). Deep learning learns features directly from data and performs feature engineering automatically, and it has been studied extensively in recommendation systems. The deep learning-based models have demonstrated significant performance over the traditional recommendation system (Singhal, Sinha & Pant, 2017). For instance, Neural collaborative filtering (He et al., 2017) utilizes a multilayer perceptron to model the interaction function as it represents users and items as a low-dimensional vector in latent space.

In another research, deep matrix factorization (DMF) (Xue et al., 2017) utilizes a matrix factorization and a neural network architecture, which uses explicit scores of users and non-preference implicit feedback of items.

The correlation denoising autoencoder (Pan, He & Yu, 2020) considers the correlation between users with diverse roles to learn a more robust representation from sparse ratings and social networks. It uses three autoencoders to learn user features taking them as a separate matrix of rating, truster, and trustee. The authors Liu et al. (2020) couple deep neural networks with matrix factorization and learn the deep global and local item relationship of item content by coupling autoencoder with matrix factorization to join the rating and item content information.

Convolutional neural networks (LeCun et al., 1998) are prevalent in image recognition, and they are generally made up of a convolution layer, pooling, and a fully connected layer. Convolutions are also used in recommender systems to model the interaction map. For instance, the convolutional factorization machine (Xin et al., 2019) is a recommender model that is context aware; it uses self-attention, an embedding layer, and a pooling layer. The authors also used an outer product interaction cube coupled with a 3D convolutional neural network to extract higher-order signals. In another research, the authors of ConvNCF (He et al., 2018) also utilize an outer product to explicitly model pairwise correlation instead of just concatenating or mere element-wise multiplication of the embedding. In addition, they also use a convolutional neural network above the interaction map to learn higher-order correlations.

Convolutional factorization machine and ConvNCF use regular convolution, which helps learn local features and does not learn global features well.

Our proposed hybrid feature selection uses deep neural networks to learn both local and global item correlation between users and items. In addition, we incorporate GMF into the model to optimize the overall model performance and prevent overfitting.

Input and embedding layer

Given a user u and item i, $V_u^U$ and $V_i^I$ represent the feature vectors of U and I respectively, and their embeddings can be represented as:

(1) $$p_u=P^T{v}_u^U,q_i=Q^T{v}_i^I$$where $P\in R^{M\times K}$ and $Q\in R^{N\times K}$ represent the embedding matrix for the user and item features, M, N, and K represent the number of users, the number of items, and the embedding size, respectively.

Interaction map

The outer product was utilized to generate the interaction map since it can learn more information between latent features. For example, the outer product between a user and an item can be defined as:

(2) $$m^t\otimes n^T=m^{t}n=\left(\begin{array}{ccc}{m}_{d_1}{n}_{d_1}& \dots & m_{d_1}{n}_{d_k}\\ ⋮& \ddots & ⋮\\ m_{d_k}{n}_{d_1}& \cdots & m_{d_k}{n}_{d_k}\end{array}\right)$$where m and n represent row vectors and denote K-dimensional latent vectors.

If $p_u=m^t$ and $q_i=n^T$ then $p_u$ and $q_i$ are used to obtain the interaction map, and it can be represented as:

(3) $$E\left(p_u,q_i\right)=p_{u\otimes }{q}_i=p_u{q}_i^T$$where $E\left(p_u,q_i\right)$ represents a $K\times K$ matrix.

Matrix factorization is not robust for modeling user-item correlation because it considers only diagonal elements and performs simple concatenation. However, the outer product is more robust and encodes more latent signals. The interaction map obtained by the outer product has one pair of latent factors, which may not perform well in the 2D convolution. As a result, we stack the interaction map into a k number of features concatenated along the dimensions. The latent signal of the interaction map determines the k-dimension features passed as input into the 2D convolution. Therefore, the increase in latent features makes the interaction map encode more relational signals, thus making it more expressive. Furthermore, the higher the number of k, the feature dimension of the input to the convolution also increases, making the model more memory and computationally intensive. However, k values greater than three do not guarantee an increase in the accuracy of the model and occasionally lead to overfitting.

Convolution module

The stack interaction map that encodes richer latent features is used as the first input to the convolution module. The convolution module is a three-layer convolution that learns the local features between users and items. The first convolution is the input layer, followed by the two hidden layers, which help to learn more meaningful information between the users and items. The input convolution layer takes the input channel of the stack interaction map, an output channel size of 32, and a kernel size of 2, while the hidden layer utilizes input and output channels of 32 with a kernel size of 2. The convolution layer is mathematically represented as:

(4) $$f_l^k\left(a,b\right)=\sum _c\sum _{p_u,q_i}{E}_c\left(p_u,q_i\right)\ast e_l^k\left(u,v\right)$$where * is the convolution, $f_l^k\left(a,b\right)$ represent the element of the feature matrix, $E_c\left(p_u,q_i\right)$ represents the element of the input stack interaction map $E_c$ of channel c, which is element-wise multiplied by $e_l^k\left(u,v\right)$ index of the kth convolution kernel $k_l$ of the lth layer. The feature map of the k^th convolutional operation can be expressed as:

(5) $$F_l^k=[f_l^k\left(1,1\right),..,f_l^k\left(a,b\right),..,f_l^k\left(A,B\right)]$$

$F_l^k$ represent the input feature matrix for the lth layers and kth neuron, A and B represent the total number of rows and columns of the feature matrix, respectively.

A convolution block or network can obtain multiple convolutional layers, dropout, and activation map for extracting meaningful information and adding non-linearity to learned complex patterns. In addition, a dropout layer is introduced to reduce overfitting, which negatively impacts the model prediction.

Hybrid feature selection module

The convolution module with a stacked interaction map which is used as input to our model, is prone to overfitting. However, it can capture relational representation well. The convolutional module introduced in “Convolution Module” is followed by a dropout layer and ReLU (Agarap, 2018) activation function. Nevertheless, the dropout minimizes overfitting at the expense of removing valuable feature relational representation. Therefore, we suggest the hybrid feature selection module (HSFM) to bridge the gap between overfitting and losing valuable information. The HSFM module takes in two inputs, y, and x, representing the output before dropout and after the ReLU activation function.

The HFSM aggregates two distinctive features to complement each other. A convenient approach to aligning two distinct feature relationships is to learn their local relationship, which can be obtained using pointwise convolutions. Therefore, the inputs x and y are summed and passed to two branches. The first branch accesses the global feature using general average pooling (GAP), and the second focuses on the local feature relationship. The two branches have two pointwise convolutions, each followed by binary normalization that minimizes the feature variation and a ReLU non-linearity. Finally, the HFSM combines global and local relationships by applying sigmoid activation on the sum of the two-branch feature, which is expressed as:

(6) $$out=x\otimes s\left(m\right)+y\otimes \left(1-s\left(m\right)\right)$$where $s$ represents the sigmoid function and m is the summation of the global and local branches.

Generally, hybrid feature selection modules access the global and local relationships from the two distinct branches that complement each other to extract better feature representation without introducing overfitting.

Generalized matrix factorization

GMF learns from data without uniform constraints and is more expressive than linear Matrix Factorization. Therefore, we combine the losses of GMF (He et al., 2017) and our proposed model to update the overall model weight, which obtains a better result and further avoids overfitting. GMF uses an element-wise product of a latent vector of users $p_u$ and items $q_i$, which can be represented as:

(7) $$z^{gmf}={\phi }_1\left(p_u,q_i\right)=p_u\otimes q_i$$

The prediction of GMF is also represented as:

(8) $${\stackrel{`}{p}}_{ui}=\sigma \left[h^T{z}^{gmf}\right]$$where $\sigma$ represents the sigmoid given as $\sigma \left(a\right)=1/\left(1+e^{-a}\right)$ and h is the weight of the output layer.

Fusion of our proposed method and GMF

Fusion in a convolutional network joins two or more features using an element-wise product, element-wise summation, or concatenation. Concatenation provides a better representation of latent features at the expense of computational complexity and memory consumption. Alternatively, features can be fused by combining the losses of different network modules, constraining the model’s overall weight by considering the submodules’ special functions.

The proposed network uses Bayesian personalized ranking (BPR), a pairwise loss function, since it measures the dependency between data points and can measure the complex relationship between data points. It can be represented as follows:

(9) $$\mathcal{L}1=\sum _{u=1}^N\sum _{i\in I_u^{+},j\in I_u^{-}}-log\sigma \left({\hat{x}}_{uij}\right)+\lambda \mathrm{\Omega }\left(\mathrm{\Phi }\right)$$where ${\stackrel{`}{x}}_{uij}=p_u^T{q}_i-p_u^T{q}_j,\sigma \left(x\right)=1/\left(1+exp\left(-x\right)\right)$ is the sigmoid function and $\lambda \mathrm{\Omega }\left(\mathrm{\Phi }\right)$ is the regularization.

On the other hand, the GMF model utilized the log loss function, which is a pointwise loss function, and it is easily computable and differentiable by the optimizer. Pointwise loss is also more flexible since it can be applied in many applications and is robust to outliers and noise in data. It can be expressed as:

(10) $$\mathcal{L}2=-\sum _{\left(u,i\right)\in {\mathcal{R}}^{+}{\mathcal{R}}^{-}}\left[f_{ui}\log f_{ui})+\left(1-f_{ui}\right)\log \left(1-{\hat{f}}_{ui}\right)\right]$$where $R^{+}$ represents positive a training instance, $R^{-}$ represents the set of negative training instances and $f_{ui}$, ${\hat{f}}_{ui}$ is the prediction and label of the GMF.

(11) $$Loss=\alpha \ast L1+\beta \ast L2$$where $L1$ is the BPR loss, $L2$ is the log loss for the GMF, and $\alpha$ and $\beta$ are the weighted coefficients.

The $\mathcal{L}oss$ combined $\mathcal{L}1,and\mathcal{L}2$ to constrain the weight of the overall proposed method, essentially avoiding overfitting and further improving the recommendation performance. In addition, we added the weighted co-efficient $\alpha$ and $\beta$ values 0.5 and 0.75, respectively, to tune the impact of the sub-networks losses for the ease of model minimization as the training epochs increase.

Final prediction layer

The output of the HFSM is reshaped and flattened using a fully connected layer to facilitate the output prediction. Finally, the result is passed into a sigmoid function to calculate the final prediction score. The array of scores ${\stackrel{`}{f}}_{ui}$ and ${\stackrel{`}{y}}_{ui}$ represent the prediction scores of GMF and the proposed CNN-based model, respectively.

Experimental settings

Performance comparison (RQ1)

Table 1 shows a comparison between our proposed model and the baselines that we used in the experiment. The performance evaluation used for the comparison utilized HR@10 and NDCG@10. In addition, for a fair comparison, we trained all the baseline models using BPR loss.

• 1) Table 1 shows that ONCF has outperformed MLP, GMF, NeuMF, DMF, and SDMR by both HR@10 and NDCG@10 on both datasets.

• 2) The CNN based on our proposed model has outperformed ONCF by a significant margin in both datasets since it does not lose much information during feature extraction to establish local relationships. In addition, The CNN-based approach utilizes stacking of the interaction map to encode better latent signals and establish a better user-item relationship.

• 3) The CNN-based approach of our proposed method does not capture the global relationship in the interaction map. Therefore, we introduced the HSFM module for the model to learn global relationships in addition to local ones. This mechanism has also improved performance over the CNN-based in HR and NDCG scores, respectively.

• 4) The fusion of GMF and our proposed method constrain the overall model weight, thus allowing the model to benefit from sub-networks. This combination obtained the best performance in our proposed methods on both datasets.

• 5) Since all the networks are trained on the BPRLoss, our proposed model performed better than the baselines on both datasets. Moreover, the proposed method obtained a remarkable performance on not only the CNN-based global and the local feature interaction for high relational modeling but the hybrid of the GMF sub-network, which shows promising results and further constrains the model weight. However, these benefits come at the expense of computation complexity.

From the charts in Fig. 1, our best-proposed model has significantly outperformed the baselines on both datasets at HR and NDCG evaluation metrics. Furthermore, among the baselines, NeuMF outperformed MLP, but it is entirely defeated by GMF, demonstrating that GMF is a simply designed yet powerful prediction model. On the other hand, NeuMF does not achieve the desired result, which may result from the selected optimizer or the poor performance of the underlying MLP in the sparse datasets.

To further evaluate the efficiency of our model, we use RMSE, MAE, and BPRLOSS and compare them to the baselines. Table 2 shows that our proposed model has effectively reduced the loss more than the baselines, thus indicating a strong minimization ability during the model training.

Efficacy of stacking the interaction map (RQ2)

The stacking of the interaction map increased the latent signal, which allowed our proposed model to establish strong user-item relationships. From Table 3, the experiments demonstrated that the increase in the stacking of the interaction map has a proportionate impact on the effectiveness of the model performance. However, beyond stack@3, the model incurred overfitting, which impaired the model performance.

Research of hyper-parameter (RQ3)

In this section, we investigated how the increased convolutional feature maps (channel size) impact the representation ability of our top-performing proposed model. As a result, we experimented with different channel sizes c ∈ {16, 64, 128}. Figure 2 shows the performance of our best model using different channel sizes on the movieLens dataset. We can observe that the increased number of channel sizes also improves the model’s performance. However, these performance gains come at the expense of increased computational complexity and are time-consuming during training.

Furthermore, the charts show that all the curves increase steadily, and the feature map at channel 128 achieves the best performance. Moreover, channel 64 steadily outperformed channel 16 until the final epochs, where a slight difference in the convergence curve was noticed. These reflect our proposed model’s strong expressiveness and generalization since increasing the number of parameters adjust the model performance and does not lead to overfitting.