An adaptive hybrid XdeepFM based deep Interest network model for click-through rate prediction system

Feature engineering

In the field of CTR prediction, the input feature dimensions are sparse, and there is no obvious spatial or temporal correlation (Shan et al., 2016; Xiao et al., 2017). The training dataset is shown as (x, y), where X mostly involves the information from user and item. Categorical fields and continuous fields are both included. Each classification field is shown as a one-hot encoding vector, and each continuous field is represented as a value itself or a discretized one-hot encoding vector. Each instance is converted to (x, y). One input instance [user id = c11, gender = female, date = Wednesday, interests = reading, ad id = book] is generally translated into high-dimension sparse features using encoding-field-aware one-hot encoding: (1)${\displaystyle \underset{\left\{user\text{\_}\left\{id\right\}=c11\right\}}{\underbrace{\left\{\left[0,\dots ,1,\dots ,0\right]\right\}}},\underset{\left\{gender=female\right\}}{\underbrace{\left\{\left[0,1\right]\right\}}},\underset{\left\{date=Wednesday\right\}}{\underbrace{\left\{\left[0,0,1,0,0,0,0\right]\right\}}},\underset{\left\{interest=study\right\}}{\underbrace{\left\{\left[0,\dots ,1\cdots ,0\right]\right\}}},\underset{\left\{a{d}_{id}=book,pencial\right\}}{\underbrace{\left\{\left[0,\dots ,1,\dots ,1,\dots ,0\right]\right\}}}}$

y ∈ 0, 1 means the associated label indicating user click behavior (y = 0 indicates that the user does not click the item, and y = 1 means otherwise). The mission of CTR prediction is to build a prediction model $\hat{y}=CTR.model\left(x\right)$ to calculate the likelihood of a user clicking on a specific app in a given situation.

Embedding layer

The embedding layer’s goal is to convert the high-dimensional binary vectors in the input into low-dimensional dense representations (Vo & Hays, 2019; Xu et al., 2017). The embedding layer is applied to the original function input, and the original data are compressed into a low-dimensional dense vector of actual values. If the field is not irresolvable, functional embedding is used as field embedding. The outcome of the embedding layer is a concatenated vector with the following format: e = [e1, e2, dotsem], where einRD signifies the embedding of one field. Different length instances can be translated into the same dimension mtimesD, where m denotes the number of fields and D denotes the field embedding dimension.

Deep interest network

When it comes to CTR prediction challenges, all deep learning systems follow the same paradigm: embedding and MLP. The large-scale sparse input features are first transferred to low-dimensional embedding vectors and, subsequently, fixed-length vectors. Finally, they are connected and fed into a fully connected layer (sometimes referred to as a multilayer perceptron or ML) to learn one of the nonlinear relationship’s properties.

From many experiments, we find that the bottleneck of expressing a user’s diverse interests in Embedding and MLP is the user representation vector with a limited dimension. Different users have diverse interests in the shopping scene, captured from users’ behavior data, which primarily affects CTR prediction. Nevertheless, in traditional methods using deep learning, nearly all of them learn to represent all user behaviors in a fixed-length vector. Unfortunately, significantly increasing the size of the learning parameters will raise the risk of overfitting with minimal data. Furthermore, it introduces the computational and storage overhead and is incompatible with online systems. Only a portion of the user’s interest will influence their behavior (it means click or not). In 2018, Alibaba proposed a deep interest network (DIN) that adaptively determines the representation vector of user interest by considering the significance of a specific candidate advertisement’s previous behavior. In the context of e-commerce apps, it pays closer attention to user behavior. DIN models this process by concentrating on the manifestation of local activation interest for a specific ad. It pays attention to user behavior in the scene of e-commerce applications. DIN simulates this process by focusing on the expression of local activation interest for a given advertisement. DIN does not use the same vector to express the different interests of all users but adaptively calculates the vector of user interests by considering the relevance of historical behavior candidate advertisements.

The architecture of DIN (Zhou et al., 2018; Zhou et al., 2019) is shown in Fig. 1. A newly designed activation unit using a neural network was obtained, and the other structures are as same as the base model. In the traditional Attention mechanism, two-item embeddings such as u and v usually obtain the dot product uv or uWv directly, where W is a weight matrix of —u—x—v—. A new solution using feature combination proposed using the shop attribute of an ad to hit the shop list of the user’s historical behavior. If it is hit, it means that the historical user has had a direct behavior. User behavior id and frequency represent this combined feature; if there is no hit, the feature is empty. The diversity of behavior data reflects users’ various interests. A user’s click of an ad often originates from only a part of the user’s interests. In the NMT task, it is assumed that the importance of each word in each decoding process is different in a sentence. The attention network can be considered a mainly constructed pooling layer that learns to assign attention ratings to each word in a phrase based on data variety.

The activation units are applied as a weighted sum pool to the user behavior features to adaptively generate the user representation, where vU of a particular candidate advertisement is designated as A. The original user behavior embedding vector and the advertising embedding vector are two parts of the activation unit’s input; the other is the vector obtained by calculating the outer product of the two embedding vectors: (2)${\displaystyle v_{u\left(A\right)}=F\left(v_A,e_1,e_2,\dots ,e_H\right)=\sum _{j=1}^{H}a\left(e_j,v_A\right)e_j=\sum _{j=1}^H{\omega }_j{e}_j.}$

The list of embedding vectors of behaviors of user U with a length of H is e1, e2, dots, eH, and vA is the embedding vector of ad A. vU (A) fluctuates in this way across distinct adverts. A () is a feed-forward network with the activation weight as the output. Aside from the two input embedding vectors, a() adds the out product to feed into the following network, explicit information to help relevance modeling. The whole structure of DIN is shown in Fig. 2.

Implicit high-order interactions

The implicit interactions (Liu et al., 2018) means the final results of the interactions are arbitrary, and there is currently no theory to demonstrate the maximum order of feature intersections that the methods—FNN, the deep part of Wide&Deep, and other CTR prediction models can learn. To learn high-order interactions, neural networks use a feed-forward neural network on the field embedding vector e. The formula for the procedure is as follows, where the layer depth is k, the activation function is sigma, and the bias is b∗. (3)${\displaystyle x^1=\sigma \left(W^{\left(1\right)}e+b^1\right)}$ (4)${\displaystyle x^k=\sigma \left(W^{\left(k\right)}{e}^{k-1}+b^k\right).}$

Explicit high-order interactions

The direct interactions of Liu et al. (2018) are the interactions whose results are straightforward and can be derived. The most famous direct high-order interaction is the DCN, whose Cross network is an improved classical fully connected feed-forward network strategy. Moreover, the hidden layer is operated by the following operation: (5)${\displaystyle x_k={x_{0}x}_{k-1}^T{\omega }_k+b_k+x_{k-1}}$ where x_k ∈ ℝ^mD is the output of the k-th layer, ω_k is the weight of the k-th layer, and b_k is the bias of the k-th layer. The Cross-layer can efficiently and explicitly learn high-order cross-features, but the problem is that a particular form limits their results, and feature crossovers are on a bitwise level rather than a vectorwise level. The unique structure of the Cross-layer allows it to display and automatically construct finite high-order feature fork multiplication.

DNN is a black-box model that implicitly interacts with high-level features. Therefore, the final result of DNN is arbitrary, and we cannot derive theoretically, nor can we obtain the maximum degree of feature interaction. Therefore, based on the idea of Deep&cross, xDeepFM was proposed in 2018 by Microsoft (Lian et al., 2018)—combining the classic plain DNN, a linear part, and a novel Compressed Interaction Network (CIN)—efficiently capturing feature interactions of bounded degrees. Unlike traditional deep neural networks that generate feature interactions implicitly at a bitwise level, xDeepFM proposed CIN, which aims to generate feature interactions at the vector level in a straightforward way. The whole structure of xDeepFM is shown in Fig. 3.

The structure is similar to Wide&Deep and DCN, and there are three parts in the xDeepFM: linear layer, CIN layer, and Deep layer. Due to this, xDeepFM has two advantages. At first, the interactions are applied at both vectorwise and bitwise levels. Secondly, it includes both low-order and high-order feature interactions. High-level feature interaction is measured, and the complexity of the network does not increase exponentially with the degree of interaction. The resulting output is as follows: (6)${\displaystyle \hat{y}=o\left(w_{linear}^{T}a+w_{dnn}^T{x}_{dnn}^k+w_{cin}^T{p}^{+}+b\right)}$ where o is the sigmoid function; a are the raw features; $x_{dnn}^k$ and p⁺ are the outputs of the plain DNN and CIN, respectively; and W∗ and b are learnable parameters.

The loss function is the negative log loss, which is the same as DIN’s loss. (7)${\displaystyle L=-\frac{1}{N}\sum _{\left(x,y\right)\mathrm{\in }s}\left(ylogp\left(x\right)+\left(1-y\right)log\left(1-p\left(x\right)\right)\right).}$

With x as the network’s input and yin0, 1 as the label, S is the training set of size N and p(x) is the network’s output, reflecting the projected probability of sample x being clicked. Furthermore, the goal of optimization is to reduce the following objective function to the smallest possible value: (8)${\displaystyle J=L+{\lambda }_{\ast }\left|\left|0\right|\right|}$ where λ_∗ represents the regularization term and λ_∗ denotes the parameters, including those in the linear, CIN, and DNN parts.

The essential point of xDeepFM is the CIN part. Cross-layer is the most refined grain with a single bit embedded in the vector, while FM is the finest-grained learning correlation, vectorwise. The motivation of xDeepFM is to introduce FM’s idea of vectorwise into the Cross-layer. The input of CIN is from the embedding layer. If there are field, and the dimension of every field’s embedding vector is D, the input will be a matrix X⁰ ∈R^m∗D and the k-th vector is calculated by this process: (9)${\displaystyle w_{h,\ast }^k=\sum _{i=1}^{H_{k-1}}\sum _{j=1}^m{W}_{ij}^{k,h}\left(X_{i,\ast }^{k-1}\mathrm{\circ }{X}_{j,\ast }^0\right)\mathrm{\in }{R}^{\mathrm{1}\ast D},where1\le h\le H_k}$ where $W_{ij}^{k,h}\mathrm{\in }{R}^{1-m\ast D}$ indicates the h-th vector weight matrix of a k-th layer. Note that X^k is derived via the interactions between X^k−1 and X⁰, so the feature interaction is measured and the degree of interaction increases with the depth of the layer.