Disentangled self-attention neural network based on information sharing for click-through rate prediction

Click-through rate prediction

Predicting whether a user will click on the recommended item is an essential problem in recommendation systems (Sangaiah et al., 2023; Guo et al., 2022; Aljunid & Huchaiah, 2022). In CTR prediction models, commonly used algorithms include linear regression (LR), gradient boosted decision tree (GBDT), FM, FFM, DeepFM, WDL, deep Interest Network (DIN), deep interest evolution network (DIEN), and others. LR served as a linear model primarily designed for sparse feature processing, while GBDT was a tree-based model proficient in managing non-linear features. FM, FFM, and DeepFM were models based on factorization machines that could handle high-dimensional and sparse features. WDL, DIN, and DIEN combined linear models and deep learning models to handle both low-dimensional and high-dimensional features. To further model automated feature interaction learning, the HoAFM (Tao et al., 2020) established intersectional features that were expressive and informative by stacking multiple cross layers. The co-action network (CAN) (Cai et al., 2021) was an effective CTR prediction model, which believed that there was no information sharing among the feature combinations of previous models, and thus introduced a dynamic pluggable feature interactive learning Unit Co-Action Unit, which realized the expression of feature combination information.

Feature interaction

The efficacy of learning feature interactions has been demonstrated in the click-through rate prediction tasks. FM was proposed mainly to capture interactions between features through factorization. Subsequently, several FM variants, such as FFM (Juan et al., 2016), AFM (Xiao et al., 2017), FmFM (Sun et al., 2021) and FwFM (Pan et al., 2018) were proposed. In recent years, some researchers have modeled higher-order feature interactions. Parallel network architecture and stacked network architecture are two classic types of click-through rate prediction models. Figure 1 shows the classic models DeepFM and NFM in two network architectures. Stacked architecture models, such as NFM (He & Chua, 2017), DIN (Zhou et al., 2018), and DIEN (Zhou et al., 2019), are representative examples. In NFM, second-order feature interactions were stacked upon deep neural networks to model higher-order features. DIN and DIEN extracted interest representations from historical behavior and utilized attention mechanisms to model the relationship between user interests and the target item.

The parallel network architecture model captures interaction signals from explicit and implicit features and then fuses the information at the output layer. Notable models in this category include DeepFM, DCN, DeepCrossing (Shan et al., 2016), xDeepFM (Lian et al., 2018), and AutoInt (Song et al., 2019). In the parallel architecture model’s implicit feature part, feature extraction primarily relied on deep neural networks (DNNs). DeepFM utilized the factorization machine (FM) structure for learning in the explicit feature interaction component. DCN proposed a cross-network approach to capture the interactions between features. The AutoInt model efficiently captured the non-linear relationships between features by adaptively learning the interaction weights between each pair of features.

Disentangled self-attention mechanism

In natural language processing tasks, attention mechanisms are extensively employed, including machine translation, text summarization, question-answering systems (Choi et al., 2016; Yang et al., 2016). The attention mechanism generally assigns weights to each input item to reflect their importance in the target task (Ali, Zhu & Zakarya, 2021). Transformers (Vaswani et al., 2017) introduced a self-attention mechanism that enabled the model to focus on different positions in the input sequence simultaneously without processing them sequentially. BERT (Devlin et al., 2018) was built by stacking bi-directional Transformer layers and achieved good performance (Wenzuixiong Xiong, 2023).

The disentangled self-attention mechanism is a method for introducing decoupling based on the self-attention mechanism. It first appeared in the field of computer vision, and the two terms were used to capture the edge and the center of an image, respectively (Yin et al., 2020). The disentangled self-attention mechanism is divided into two parts: the paired term and the unary term. Paired terms are used to model a specific interaction between two features, and unary terms are used to model the effect of one feature on the other. DESTINE computed higher-order feature interactions by stacking multiple disentangled self-attention layers. It does not allocate the features of the unary term to distinct subspaces. In addition, we think that the stacked network architecture is essentially a linear model, which may not capture the higher-order interaction relationships between more complex features, limiting the expressive power of the model.

Problem definition

The main objective of CTR prediction is to assist advertisers or e-commerce companies in estimating the likelihood of users clicking on recommended items. It helps select the most valuable items for recommendation in the recommendation system, thereby maximizing its effectiveness and commercial value. Let us assume that the entire dataset $D=\left\{\left(x_1,y_1\right),\left(x_2,y_2\right),\dots ,\left(x_N,y_N\right)\right\}$ consists of N examples, where each sample x_i consists of M user and item feature fields, and its associated label y_i ∈ {0, 1} is the ground truth ith sample. CTR prediction is the probability of a user clicking on a recommendation using the formula $\hat{y}=f\left(x_i\right)$ under the given feature vector x_i.

Embedding layer

In vision or natural language processing, the input data is often images or text signals with spatial or temporal relevance. However, input features in the recommendation system are usually sparse, and there is no apparent spatiotemporal correlation. Because CTR predictions contain discrete categorical variables that cannot be directly used for numerical computations, feature embedding is essential to forecasting click-through rates. Suppose the entire dataset consists of N examples, each sample consisting of M user and item feature fields. Some of these features are categorical data, while others are numerical data. The most often used approach for categorical data is feature embedding, which involves converting each sparse vector into a low-dimensional dense vector. To illustrate, if the one-hot vector of the ith field is designated as x_i, the related embedding matrix is written as w_i. The formulation of the e_i can be represented as follows. ${\displaystyle e_i=w_i{x}_i}$ where x_i is a one-hot vector and w_i is the embedding matrix of the ith sparse feature. By applying a conversion process, the numerical feature x_j can also be transformed into the same low-dimensional space. ${\displaystyle e_j=w_j{x}_j}$ where w_j is an embedding vector and x_j is a scalar value.

According to the above method, the embedding layer compresses a high-dimensional sparse vector into a low-dimensional dense vector. The equation for the embedding layer is expressed as: ${\displaystyle E=\left[e_1;e_2;\dots e_i;\dots e_j;\dots e_M\right]}$ where “;” represents the matrix stacked operation, and M represents the number of feature fields.

Disentangled self-attention layer

Self-attention is a mechanism that enables the model to assign varying degrees of importance to different positions in the input sequence, facilitating accurate predictions. In traditional self-attention, a single attention head is used to compute the attention weights between all pairs of positions in the input sequence. In multi-headed self-attention, the input sequence is mapped to multiple attention subspaces separately, and independent self-attention computations are performed in each subspace. Each attention head can learn different attention patterns and dependencies to capture multiple levels of information in the input sequence. Building upon the principles of multi-head self-attention, this article introduces the disentangled multi-head self-attention layer, which is further divided into paired terms and unary terms.

For the input feature x_i, it is transformed into a dense embedding vector e_i by embedding search. After obtaining the low-dimensional representation of each feature, We use the dot product attention approach to simulate higher-order interactions between features. We define four different matrices $w_q^{\left(h\right)}$, $w_{k1}^{\left(h\right)}$, $w_{k2}^{\left(h\right)}$ and $w_v^{\left(h\right)}$, then multiply e_i by each of them, and the four vectors are represented as follows: ${\displaystyle \left\{\begin{array}{c}{Q}_i^{\left(h\right)}=w_q^{\left(h\right)}{e}_i\hfill \\ K_i^{\left(h\right)}=w_{k1}^{\left(h\right)}{e}_i\hfill \\ {\tilde{K}}_i^{\left(h\right)}=w_{k2}^{\left(h\right)}{e}_i\hfill \\ V_i^{\left(h\right)}=w_v^{\left(h\right)}{e}_i\hfill \end{array}\right.}$ where $Q_i^{\left(h\right)}$, $K_i^{\left(h\right)}$, ${\tilde{K}}_i^{\left(h\right)}$ and $V_i^{\left(h\right)}$ are obtained by the four weights $w_q^{\left(h\right)}$, $w_{k1}^{\left(h\right)}$, $w_{k2}^{\left(h\right)}$ and w_v ∈ ℝ^d′×d respectively. d is the dimension of the filed embedding, and d′ is the dimension of the attention.

It has been demonstrated in previous visual learning tasks that the standard self-attention mechanism is detrimental to feature learning. The disentangled self-attention mechanism is used in this article, and paired and unary terms are disentangled utilizing activation functions and embedding matrices. For the paired terms, in order to eliminate the offset information between features and enhance the expressiveness of the model, we obtain the multi-head feature representation of $Q_i^{\left(h\right)}$ and $K_i^{\left(h\right)}$, and then subtract their average values. Similar to the ordinary self-attentive mechanism, the two feature representations above are multiplied together and then passed through the softmax activation function to obtain the vector of attentional weights $P_i^{\left(h\right)}$ for the paired terms. For the unary terms, we first map the features into different subspaces, from which multiple feature representations are learned to obtain the matrix ${\tilde{K}}_i^{\left(h\right)}$. Then, the mean value ${\mu }_{k2}^{\left(h\right)}=\frac{1}{M}{\sum }_{i=1}^M{w}_{k2}^{\left(h\right)}{e}_i=\frac{1}{M}{\sum }_{i=1}^M{\tilde{K}}_i^{\left(h\right)}$ of ${\tilde{K}}_i^{\left(h\right)}$ is calculated and multiplied by the matrix $K_i^{\left(h\right)}-{\mu }_{k1}^{\left(h\right)}$ to obtain the characteristic representations. Finally, the attentional weight vector $U_i^{\left(h\right)}$ is obtained by the softmax activation function. After we have the paired and unary terms, we add the two terms and dot them with $V_i^{\left(h\right)}$. The calculation formula is as follows: ${\displaystyle \left\{\begin{array}{c}{P}_i^{\left(h\right)}={\sigma }_4\left({\left(Q_i^{\left(h\right)}-{\mu }_q^{\left(h\right)}\right)}^T\left(K_i^{\left(h\right)}-{\mu }_{k1}^{\left(h\right)}\right)\right)\hfill \\ U_i^{\left(h\right)}={\sigma }_5\left({\mu }_{k2}^{\left(h\right)}{\left(K_i^{\left(h\right)}-{\mu }_{k1}^{\left(h\right)}\right)}^T\right)\hfill \\ {Head}^{\left(h\right)}=\sum _{i=1}^M\left[U_i^{\left(h\right)}+P_i^{\left(h\right)}\right]V_i^{\left(h\right)}\hfill \end{array}\right.}$ where σ₄(⋅) and σ₅(⋅) are the softmax activation function, ${\mu }_q^{\left(h\right)}=\frac{1}{M}{\sum }_{i=1}^M{w}_q^{\left(h\right)}{e}_i$ and ${\mu }_{k1}^{\left(h\right)}=\frac{1}{M}{\sum }_{i=1}^M{w}_{k1}^{\left(h\right)}{e}_i$ take average of the $Q_i^{\left(h\right)}$ and the $K_i^{\left(h\right)}$ vectors, respectively, and M is the total number of features for users and items. Then, we connect all the attention heads with the following formula: ${\displaystyle Z=\left[{Head}^{\left(1\right)};{Head}^{\left(2\right)};\dots ;{Head}^{\left(h\right)}\right]w_1+b_1}$ where h is the number of attention heads, and Z stacks up all the features after getting the attention head. w₁ means the weight matrix, and b₁ denotes the bias.

To preserve the information contained within the original embedded vector, a residual structure is incorporated into the network. Furthermore, normalization and ReLU activation function procedures are implemented before the output feature representation to increase the model’s stability, convergence speed, and performance. ${\displaystyle L_o=\phi \left(Norm\left(Z+w_{r}E\right)\right)+b_r}$ where w_r ∈ ℝ^d′×d is a linear projection matrix to avoid dimension mismatch, φ(⋅) is the ReLU activation function.

Sharing interaction layer

There are two primary modules in the shared interaction layer: the decomposition module and the sharing module. The decomposition module distinguishes feature distribution in different networks by field control networks. The shared module establishes a link between the hierarchical attention and the deep network and captures the interaction signals between the two networks. These two modules are lightweight with minimal time and space complexity, and they can be easily applied to the CTR model of parallel network architecture.

Output layer

The loss function is utilized in the click-through rate prediction task to calculate the difference between the model forecast and the actual click-through rate situation. The model output is a probability value between 0 and 1, and the real value can only be 0 or 1. Binary cross-entropy measures the loss by calculating the difference between the predicted values and the actual values. The loss function seeks to minimize cross-entropy in the training process so that the predicted result can accurately match the actual click situation. Our loss function is J, which is defined as follows:

${\displaystyle J=-\frac{1}{N}\sum _{i=1}^N\left(y_{i}log\left({\hat{y}}_i\right)+\left(1-y_i\right)log\left(1-{\hat{y}}_i\right)\right)}$ where y_i and ${\hat{y}}_i$ represent the real value and predicted value, respectively, and y_i is the true label of the ith sample. We use a gradient descent algorithm to update model weights.

Experimental settings

Performance comparison

In this section, we report the model’s performance on two datasets. The DSAN and the best baseline are emphasized in bold and underlined formats. The “Previously Reported” column shows the best results for both datasets that we found in our existing work. The “#Params” indicates the number of parameters used in each model. Table 2 presents the performance of models, and we can draw the following conclusions:

• In the “Previously Reported” column, the performance of InterHAt is inferior to LR on both datasets. It could be attributed to differences in how the models preprocess the datasets and the splitting ratios. This article evaluates all models using the same evaluation protocol and preprocessing methods for result comparability. In addition, we perform a paired t-test to verify the statistical significance of the relative improvement of DSAN.

• The performance obtained on Avazu datasets is essentially the same or even higher than the reported results. For example, InterHAt and DCN receive better results on Avazu datasets. The experimental results show that LR and FM shallow models perform worse because they cannot catch complex higher-order feature interactions. Deep models are more advantageous in capturing complex higher-order feature interactions than shallow models and usually perform better. Models like WDL, NFM, and DeepFM, among others in deep learning, demonstrate superior performance compared to shallow models. By adjusting the parameters, we find that the differences between most models become minimal. The discrepancies between DCN and DeepFM on the Avazu dataset are negligible. During the model training process, we have noticed that most models exhibit a phenomenon called one-epoch overfitting, meaning that training for just one epoch typically suffices to achieve optimal performance. According to the relevant literature (Zhang et al., 2022), it points out that feature sparsity is the cause of one epoch. Improving model performance by improving data sparsity may be a worthwhile research topic.

• The performance of the Criteo dataset is generally in line with the reported results, but in some aspects, it is slightly lower than expected. The lack of detailed hyperparameter information provided in the article resulted in us not obtaining the best combination of hyperparameter tuning and feature engineering. However, InterHAt and NFM models demonstrate superior performance on Criteo datasets, underscoring the effectiveness of our approach to data preprocessing and hyperparameter optimization. We can also observe that the Criteo dataset has better feature representation at an embedding dimension of 16, while the Avazu dataset obtains better results at higher embedding dimensions. This difference may stem from the complexity and characteristics of the datasets themselves, leading to variations in the optimal choice of embedding size.

• Tuning model parameters is one of the crucial steps in deep learning. We adjust the models regarding embedding size, number of attention heads, and network layers. We tune the embedding size to { 16, 24, 32, 40, 48 } for both datasets to investigate the effect of embedding dimensions on the model. The number of network layers in deep learning is tuned between 0 and 4. To further illustrate the need for tuning, Table 3 shows the results of the three benchmark models before and after tuning. The “Reported” reflects the findings of the existing research; “Rerun” denotes the outcome pre-adjustment; and “Retuned” signifies the outcome post-adjustment. We can observe that while DeepLight and FinalMLP didn’t achieve the reported performance on the Criteo dataset, they slightly surpassed the previous performance on the Avazu dataset. This variance might arise from the distinct characteristics and complex relationships among features in these datasets. It indicates that the characteristics of the data influence a model’s performance across different datasets, and it requires specific adjustments tailored to each dataset to achieve better performance.

Performance analysis

In this section, we examine the performance of the model. We compare the proposed disentangled self-attention layer with the traditional self-attention approach and investigate the performance of three variants of the sharing module.

Disentangled self-attention layer performance analysis

The disentangled self-attention mechanisms are a method that introduces decoupling based on self-attention mechanisms. We devise two comparative approaches to assess the efficacy of the disentangled self-attention mechanism proposed in this article, and we compare it with the disentangled self-attention layer in DESTINE. Method one, DSAN alt SA, combines the common self-attention mechanism with the shared interaction layer. The second method, DSAN alt DS, combines the disentangled self-attention mechanism proposed in Xu et al. (2021) with the shared interaction layer.

According to the experimental results in Fig. 3, our method DSAN outperforms the two methods above on both datasets. Compared with DSAN alt SA, the AUC of this article improved by 0.9% and 0.6% on the Criteo and Avazu datasets, respectively. The results show that the disentangled self-attention mechanism proposed in this article is superior to the common self-attention mechanism. Compared with DSAN alt DS, DSAN improved the AUC by 0.6% and 0.3% on the two datasets, Criteo and Avazu, respectively. This result indicates differences in the design of disentangled self-attention layers between DESTINE and this article. In this article, we successfully capture the differences and variation range between the unary term characteristics of the disentangled self-attention layer, thus improving the model’s performance.

Table 3:

Results before and after hyperparameter tuning.

Model	Setting	Criteo		Avazu
		AUC	LogLoss	AUC	LogLoss
InterHAt	Reported	0.7845	0.4577	0.7582	0.3910
	Rerun	0.7961	0.4528	0.7618	0.3857
	Retuned	0.8016	0.4497	0.7886	0.3749
DeepLight	Reported	0.8116	0.4403	0.7893	0.3751
	Rerun	0.8087	0.4430	0.7887	0.3759
	Retuned	0.8092	0.4423	0.7893	0.3750
FinalMLP	Reported	0.8149	–	0.7666	–
	Rerun	0.8069	0.4461	0.7762	0.4016
	Retuned	0.8073	0.4442	0.7849	0.3771

DOI: 10.7717/peerjcs.1764/table-3

Different variant of sharing module

We investigate the effects of three different variants in the shared module. To capture the signal between different networks of the parallel architecture, we express the Hadamard Product of two vectors as DSAN-HP, which is also the way used in the article, and thus as DSAN. The inner product and Hadamard Product feature fusion are denoted as DSAN-IH, and the way the two vectors are connected and passed through the feed-forward neural network is denoted as DSAN-CN. As shown in Fig. 4, DSAN performs the best. It is due to the Hadamard Product being in the same position as the elements and not involving any weights or coefficients.

Ablation study

We conduct experiments to verify the efficacy of individual components within the DSAN model and understand their respective significance. Each experiment entails the removal of a single component while keeping the remaining components unchanged, ensuring a focused evaluation of their relative importance. The DSAN w/o DA denotes the removal of disentangled multi-head self-attention, implying only a shared interaction layer in the model. DSAN w/o PT is the removal of paired terms in disentangled multi-head self-attention, and DSAN w/o UT is the removal of unary terms in disentangled multi-head self-attention. DSAN w/o SI indicates the removal of the part of the two modules that share the interaction layer, which means that only the DNN network and the attention aggregation exist in the parallel structure.

Based on Table 4, it is evident that eliminating any component from DSAN results in a decline in model performance. Compared to DSAN w/o DA, DSAN improves the AUC values by 1.2% and 0.9% on both the Criteo and Avazu datasets, respectively. This progress demonstrates the efficacy of the introduced disentangled self-attention layer in bolstering the model’s accuracy. To verify the effectiveness of the unary and pairwise terms presented in the disentangled self-attention layer, we design DSAN w/o PT and DSAN w/o UT. Compared to the DSAN w/o SI, the DSAN model exhibited 1.2% and 0.6% improvements in AUC values on the Criteo and Avazu datasets, respectively. This demonstrates that integrating decomposition and sharing modules into a parallel network architecture is effective.