Autoregressive models for session-based recommendations using set expansion

Input and embedding layer

In the model’s input process, to prevent the interaction order from affecting the prediction results, we use a set representation when processing session context information. Specifically, we construct the model’s input as an interaction set $\mathbf{s}=\{v_1,v_2,\cdots ,v_m\}$, where the elements $v_i$ in the set correspond to the interaction items in the original session. By using a set representation, the model’s permutation invariance is ensured, thereby guaranteeing that the output of the embedding layer is not influenced by the order of the input sequence (Wagstaff et al., 2022).

When the input interaction set $\mathbf{s}$ passes through the model’s embedding layer, the context information received by the embedding layer consists only of discrete item IDs, which is insufficient for set learning and prediction tasks. To capture the complex relationships between items, we employ a learnable embedding matrix $\mathbf{W}\in {\mathbb{R}}^{d\times \left|I\right|}$, where $d$ is a learnable hyperparameter dimension, and $\left|I\right|$ is the size of the vocabulary. Each row of the embedding matrix $\mathbf{W}$ corresponds to an item, and this matrix maps the ID information into a low-dimensional, dense continuous vector space. Through this embedding process, we obtain the embedded interaction set $\mathbf{X}\in {\mathbb{R}}^{m\times d}$, where the embedding layer can be represented as follows:

(2) $$x_i=\left\{{\mathbf{W}}_{:,i}\right\}.$$

Here, ${\mathbf{s}}_{1:m}$ represents the set of items from the first to the $m$th item in the session, and $f_{rec}$ is the set-based learning algorithm.

Permutation invariance modeling

In traditional session-based recommendation models, recurrent neural networks (RNNs) or graph neural networks (GNNs) are often used to model interaction sequences. However, these methods typically rely on the chronological order of interactions, which can lead to suboptimal recommendation results in certain cases. To overcome this limitation, this study adopts the permutation-invariant design concept from the Deep Sets model (Zaheer et al., 2017), treating the interaction sequence as a set and modeling it through set learning.

Specifically, suppose the input session is a set $\mathbf{s}=\{v_1,v_2,\dots ,v_n\}$, where each element $v_i$ is an item in the interaction sequence. To ensure the model’s permutation invariance—that is, the output remains unchanged when the order of the input sequence is altered—the Deep Sets model applies the same mapping operation to each element in the set and then combines these mapped results through an aggregation function. This process can be expressed as:

(3) $$\varphi (s)=\rho (\sum _{i=1}^n\varphi (v_i))$$where the mapping function $\varphi$ is a universal approximator, and the aggregation function $\rho$ aggregates the mapping results of all elements.

According to the Deep Sets theory (Zaheer et al., 2017), as long as the aggregation function is symmetric (such as summation, averaging, etc.), the model can remain invariant to the input order. This design allows the model to focus on capturing the co-occurrence features within the set without relying on the temporal order of the items.

Set extension

As described in “Model Overview”, the set extension module focuses on extracting co-occurrence information from the session embedding vectors $\mathbf{X}$ to capture the overall co-occurrence features of the session (Zhao et al., 2023). The embedding vectors $\mathbf{X}$ form a set of $d$-dimensional elements. Based on the model design in “DSETRec Model”, the model learns the entire set during the set learning layer while maintaining permutation invariance (Lee et al., 2019). Permutation invariance refers to the property that, if there is a function $f$ that operates on the set $\mathbf{X}$, the result of the function is unaffected by the order of elements in the set (Zaheer et al., 2017).

This means that in our model’s set learning layer, the learning outcome remains unaffected by the order in which the items are processed, which can be expressed as:

$$\begin{array}{c}(p_1,p_2,\cdots ,p_n)\ne (q_1,q_2,\cdots ,q_n)\hfill \\ SL(\left\{v_{p_1},v_{p_2},\cdots ,v_{p_n}\right\})=SL(\left\{v_{q_1},v_{q_2},\cdots ,v_{q_n}\right\})\end{array}$$where $(p_1,p_2,\cdots ,p_n)$ and $(q_1,q_2,\cdots ,q_n)$ represent two different permutations, and SL is the set learning algorithm.

This characteristic ensures that when the input embedding vectors $\mathbf{X}$ are presented, the set learning layer can capture the same features regardless of the order of the data. According to research (Zaheer et al., 2017), if a function $f:\mathcal{X}\to \mathcal{Y}$ has permutation invariance, it is necessary and sufficient that the function can be decomposed as $\rho \left({\sum }_{x\in X}\phi (x)\right)$, where $\rho$ and $\phi$ can represent arbitrary functions or universal approximators.

The overall structure of the set learning is shown in Fig. 3. The core of set extension lies in effectively extending the existing set to predict the missing elements in the set. For session-based recommendation tasks, this means predicting the next possible interaction item $v_{n+1}$ based on the historical interaction item set $\mathbf{s}=\{v_1,v_2,\cdots ,v_n\}$. The set extension strategy based on Deep Sets can be implemented as follows.

First, we map and aggregate the input set $\mathbf{s}$: Considering that each stage can be viewed as an independent set and that the order of elements in these sets should not affect the overall prediction results, we incorporate the concept of permutation invariance into the model and model it. Let $X_1,X_2,\dots ,X_n$ represent the input sets at each stage, and the model must ensure that when any permutation $\pi$ is applied to the input sets, the output remains unchanged. Specifically, we can define the permutation-invariant function as:

(4) $$f(X_1,X_2,\dots ,X_n)=\rho (\sum _{i=1}^n\varphi (X_i))$$where $\varphi$ is a feature extraction function for each set element, and $\rho$ is a nonlinear transformation used to aggregate the multi-stage results. Through this architecture, we can effectively achieve permutation-invariant modeling of multi-stage inputs, ensuring the model’s stability when faced with inputs in different orders.

When processing co-occurrence information between set elements, we successively use two deep neural networks as universal approximators to complete the set learning task. To ensure that this set-based method can be widely applied to various session-based recommendation tasks, we used a relatively simple fully connected neural network as our approximator, although any deep neural network can be substituted. During the set learning layer processing, the input vectors $\mathbf{X}$ are first mapped to a new space by a universal approximator $\phi$ for each element in the set, and then these mapped elements are aggregated through an aggregation operation. The resulting output is further mapped through a second approximator $\rho$ for the next step.

In practice, the universal approximator $\phi$ chosen in this study is a stack of two fully connected neural networks. During the processing of these neural networks, we also introduced two residual connections (He et al., 2016) to further enhance the extraction of co-occurrence features from the elements in the set by the universal approximator and to prevent potential issues of vanishing or exploding gradients. The choice of different aggregation methods following the approximator $\phi$ can also affect the model’s convergence speed, which we further discuss in the experimental section.

In the universal approximator $\phi$, we also use an aggregation operation, denoted as $\varrho$. We can express the universal approximator $\phi$ as follows:

(5) $$\begin{array}{c}DN{N}_i={\mathbf{W}}_{i}F+{\mathbf{b}}_i\hfill \\ \phi =DN{N}_2\left(\varrho \left(DN{N}_1\left(\mathbf{X}\right)+\mathbf{X}\right)\right)+\varrho \left(DN{N}_1\left(\mathbf{X}\right)+\mathbf{X}\right))\end{array}$$where $DN{N}_i$ represents the $i$th deep neural network in the universal approximator $\phi$, and ${\mathbf{W}}_i$ and ${\mathbf{b}}_i$ represent the corresponding parameter matrix and bias vector, respectively. $\varrho$ is the aggregation operation.

After passing the input set $\{x_1,x_2,\dots ,x_n\}$ obtained from the embedding layer through the universal approximator $\phi$, it can be transformed into $\phi (x_1),\phi (x_2),\dots ,\phi (x_n)$. We then continue by passing these through the aggregator $\rho$, which includes the aggregation operation $\varsigma$ and a second neural network.

(6) $$\begin{array}{c}\rho ={\mathbf{W}}_{j}F+{\mathbf{b}}_j\hfill \\ F=\rho \varsigma (\phi (x_1),\phi (x_2),\cdots ,\phi (x_n))\end{array}$$where $\varsigma$ represents the aggregation operation, including summation or averaging aggregation operations. $\rho$ is the aggregator composed of a fully connected network, and ${\mathbf{W}}_j$ and ${\mathbf{b}}_j$ are its corresponding parameter matrix and bias vector.

The aggregator differs from the approximator in that it implements changes to the entire set that has been mapped by the approximator. It is worth noting that the aggregator also needs to satisfy the “symmetric design,” meaning it should avoid the influence of the input order of elements on the set. Therefore, we use summation or averaging as the aggregation methods here, and we further discuss the performance impact of different aggregation methods in subsequent experiments. After passing through the aggregator $\rho$, we obtain the aggregation vector F and consider it to contain the co-occurrence features of the session <D>, which we continue to use in the prediction layer (Wu et al., 2020).

Finally, we use the prediction head to transform the aggregated co-occurrence information into probability output. The prediction head is essentially a linear layer, and this mapping operation consists of a weight matrix and an optional bias term. After the interaction items in the session $\mathbf{s}$ pass through the set learning layer to obtain the final output F, we consider that F contains the co-occurrence features learned from the current session. The prediction head maps the output F, containing the co-occurrence features, to a dimension corresponding to the total item set $\mathbf{V}$ and ultimately outputs the probabilities through the softmax function.

(7) $$\hat{\mathbf{Y}}=softmax({\mathbf{W}}_{p}F+{\mathbf{b}}_p)$$where ${\mathbf{W}}_p$ and ${\mathbf{b}}_p$ are the parameter matrix and bias vector of the prediction head. We share the weights between the embedding layer and the prediction head to alleviate overfitting and reduce model size.

At this point, our model successfully maps the co-occurrence features in the interaction set $\mathbf{s}$ into recommendation probabilities in the total item set $\mathbf{V}$ and outputs the probability list $\hat{\mathbf{Y}}={\hat{y}}_1,{\hat{y}}_2,\dots ,{\hat{y}}_{\left|v\right|}$, which contains the predicted scores for all possible interaction items in the total item set $\mathbf{V}$. From the probability list $\hat{\mathbf{Y}}$, we select the interaction item $v_i$ corresponding to the highest predicted score ${\hat{y}}_i$, which is the next recommended item $v_{m+1}$ for the session.

Multi-stage autoregressive prediction

In the DSETRec model, autoregressive prediction is achieved by gradually extending the current session set. Specifically, we first apply a masking operation to the current session $\mathbf{s}=\{v_1,v_2,\dots ,v_n\}$ to generate a partially observable set $S^{\mathrm{\prime }}$. Then, using the set extension method, we predict the missing items in the set and progressively add them to the current set.

Let $S^{(t)}$ represent the state of the session set at step $t$, then the autoregressive prediction can be expressed as:

(8) $${\mathbf{S}}^{(t+1)}={\mathbf{S}}^{(t)}\cup v^{(t+1)}$$where $v^{(t+1)}$ is the next item predicted by the set extension method: $v^{(t+1)}=\arg \underset{v\in V}{max}P(v|{\mathbf{S}}^{(t)})$. This process iterates until a complete recommendation sequence is generated.

During the prediction phase, for each stage’s input, we introduce a masking operation that progressively hides parts of the input, forming an autoregressive chain structure. Specifically, let the input set at the current stage be $X_m$; the model applies a masking operation M to generate partially observed data:

(9) $${\mathbf{X}}_m=\mathbf{M}({\mathbf{X}}_m).$$

Then, the set extension mechanism is used to predict the remaining parts of the set:

(10) $${\mathbf{X}}_m=\psi ({\mathbf{X}}_m)$$where $\psi$ represents the set extension prediction function. This process relies on the results of the previous stage and iteratively updates according to the following autoregressive formula:

(11) $${\mathbf{X}}_{m+1}=g({\mathbf{X}}_m,{\mathbf{X}}_{m-1}).$$

At each stage, by stacking nonlinear activation functions and attention mechanisms, the model enhances its ability to capture long-term dependencies. In particular, the attention mechanism can weigh and sum the input through the weight matrix $A_m$:

(12) $${\mathbf{Z}}_m={\mathbf{A}}_m\cdot {\mathbf{X}}_m.$$

Through the above steps, this method not only effectively reduces information redundancy in multi-stage prediction but also improves the model’s stability and accuracy when handling long sequences.

Additionally, the loss function used during the training of the model is Top 1 loss (Hidasi et al., 2015). This loss function allows the model to select only the detection result with the highest confidence as the positive sample, while other detection results are treated as negative samples. Previous studies and our experiments have demonstrated the effectiveness of this loss function (Hidasi & Karatzoglou, 2018). The loss function is as follows:

(13) $$L_{top1}=\frac{1}{N_s}\sum _{j=1}^{N_s}\sigma \left({\hat{r}}_{s,j}-{\hat{r}}_{s,i}\right)+\sigma \left({\hat{r}}_{s,j}^2\right).$$

Comparison of performance between set data and sequential data

To further validate our proposed hypothesis, this section compares the performance of the DSETRec model with position encoding, representing “ordered” data (Romero & Cordonnier, 2020), against the DSETRec model without position encoding. Comparative experiments were conducted on the two commonly used datasets mentioned earlier, and the results are shown in Table 2. We have highlighted the values where the performance of the ordered and unordered models is relatively better in bold.

As shown in the table above, when using the DSETRec model, the unordered set model without position encoding consistently outperforms the ordered sequence model with position encoding across all datasets. This indicates that our DSETRec model is more adept at handling set-type data.

For the session-based recommendation tasks on the above datasets, the DSETRec model may not be sensitive to the positional information of the interaction items. In practical recommendations, the order of item interactions within the same session can be arbitrary and incidental, with the interaction order not necessarily guiding the recommendation results. Therefore, introducing position encoding does not provide the model with critical information and may even introduce noise, reducing the model’s performance. The presence of position encoding, representing “order,” hinders the aggregation of co-occurrence information among session items. This “hindrance” effect is particularly evident in the more easily predictable Yoochoose dataset (Chen, Li & Zhou, 2021).

Model performance analysis

To further demonstrate the performance of the set-based learning recommendation algorithm proposed in this article, we compared the DSETRec model with other recommendation algorithms or models. To ensure a fair comparison, the Baseline methods retained the original data preprocessing, filtering out items with fewer than five interactions. The results are shown in Table 3. In order to better observe the performance comparison with the baselines, we have highlighted the results of the DSETRec model in bold and underlined the best-performing results.

When comparing with other algorithms or models, it is clear that the DSETRec model proposed in this article outperforms all the classical recommendation algorithms and most deep learning models mentioned, providing strong validation for the effectiveness of our proposed recommendation algorithm. However, despite this, our model shows some gaps in Precision compared to some state-of-the-art models like SR-GNN. Nevertheless, in terms of the MRR metric, our model performs exceptionally well, surpassing all other leading models in the field. This could be attributed to our model’s training approach, which applies each item more frequently across the dataset than the training methods of other models, playing a crucial role in capturing the co-occurrence features of the session.

Overall, the DSETRec model, based on a general architecture, demonstrates strong performance and provides preliminary validation for the feasibility of set-based learning in session-based recommendation. This also underscores the importance of session-wide co-occurrence features, which contribute more significantly to recommendation effectiveness compared to the sequential features among items.

Long and short session performance analysis

To validate the generalizability of our model across sessions of different lengths, we conducted performance comparison experiments on the DSETRec model for sessions of varying lengths. We used the total dataset, which includes all sessions, as the performance benchmark. Sessions with five or fewer items were defined as short sessions, while those with more than five items were defined as long sessions. To further evaluate the model’s adaptability to different session lengths, we chose $K=20$ as a key configuration. This choice is based on the fact that extremely short sessions (e.g., fewer than five items) may lack sufficient information, resulting in reduced model performance when understanding user behavior and making accurate recommendations. On the other hand, excessively long sessions may contain redundant information, increasing the model’s computational complexity and potentially overwhelming it with unnecessary details. By selecting $K=20$, we aimed to strike a balance between leveraging enough contextual information and avoiding performance bottlenecks from overly short or long sessions. With $K=20$, we applied the DSETRec model separately to the total sessions, long sessions, and short sessions, and compared the gap in Precision between the long and short session datasets. The performance of the model on sessions of different lengths is shown in Table 4.

As seen in the table above, the performance of the DSETRec model on the long session dataset only decreased by about 5% compared to its performance on the overall dataset, indicating that it still maintains good effectiveness in handling long sessions. Comparisons with other models (Liu et al., 2018; Li et al., 2017) show that our model exhibits the highest stability, further confirming our hypothesis for session-based recommendation.

Hyperparameter sensitivity analysis

In this section, we analyze the impact of hyperparameters on the performance of the DSETRec model. To more intuitively observe the model’s performance under different hyperparameter settings, we selected an unfiltered dataset for this experiment. By adjusting the model’s dimension and batch size, we conducted corresponding experiments to explore their effects on precision (P) and mean reciprocal rank (MRR). With $K=20$, the results of the analysis experiments are shown in the Fig. 4.

Regarding the dimensionality, our model tends to prefer higher dimensions. Higher dimensionality allows the model to learn co-occurrence information from different subspaces, which promotes the aggregation of co-occurrence features. It is worth noting that the improvement in model performance is not proportional to the increase in dimensionality. Our model reaches near saturation at 256 dimensions, with further increases in dimensionality providing very limited gains. For the Diginetica dataset, this saturation effect is even more pronounced, with the P and MRR metrics showing different levels of decline at 512 dimensions. As shown in Fig. 5, compared to model dimensionality, batch size has a relatively smaller impact on the model’s performance, showing an overall trend of initial decline followed by a slow rise. Our model tends to perform better with smaller batch sizes. A preliminary analysis suggests that this may be because our gradual training approach during data training is somewhat similar to batch processing, leading to a degree of redundancy that affects certain aspects of performance.

Aggregation method analysis

In our model, we employed two aggregation operations (the aggregation operation $\varrho$ and the aggregation performed by the aggregator $\rho$). Common aggregation methods include summation, averaging, and taking the maximum or minimum values. The choice of aggregation method in the experiments had a noticeable impact on our model’s performance. The results for different aggregation methods are illustrated in the Fig. 6.

In the figure, AVG + SUM indicates that averaging aggregation was applied first, followed by summation aggregation, while AVG++ indicates that averaging was used for both aggregation steps, and so on. The figure shows that using averaging first, followed by summation, provides the best performance for our model. Specifically, for the aggregation operation $\varrho$, averaging helps capture the details within the input vectors $\mathbf{X}$. For the aggregator $\rho$, which is at the final stage of the model, summation is effective in retaining more co-occurrence information.

Ablation study

To further explore the effectiveness of the autoregressive and set extension modules in our proposed model, we conducted ablation experiments on the Yoochoose and Diginetica datasets. The results are shown in Fig. 7.

The ablation study results demonstrate that removing either the set extension module (w/o set extension) or the autoregressive module (w/o autoregressive) leads to a noticeable decline in the model’s performance across both P@20 and MRR@20 metrics. Among the two, the absence of the set extension module has a more significant impact on the results, indicating its critical role in enhancing the recommendation quality.

This suggests that the set extension module is essential for capturing complex user interaction patterns, while the autoregressive module contributes to the sequential dependency modeling. The complete DSETRec model consistently achieves the best performance, underscoring the effectiveness of the designed architecture and the complementary contributions of these two modules.