A user-embedded temporal attention neural network for IoT trajectories prediction

Motivation

In the age of the Internet of Things (IoT), an immense volume of data is produced by countless devices, surpassing the processing capabilities of traditional data processing services. To extract maximum value from this massive amount of data, time series analysis is essential. Time sequences and time series data are closely related, necessitating the development of advanced algorithms that can efficiently process and analyze these large datasets.

Problem statement

Sequential recommendation has gained significant research interest over the past two decades due to its potential applications in commodity recommendation. Traditional methods such as the Markov model, which relies on a state transition matrix for forecasting future behavior, face challenges in capturing intermittent transfers due to sparse sequence data. With the advent of deep learning, models based on recurrent neural networks (RNNs) and long short-term memory (LSTM) (Hochereiter & Schemidhuber, 1997; Gers, Schmidhuber & Cummins, 2000) networks have shown superior performance in handling sequential data. However, LSTM-based models face limitations in parallel computing, leading to the widespread adoption of attention mechanisms for sequence prediction.

Recent advancements in attention-based (Vaswani et al., 2017) models, such as self-attention based sequential model (SASRec) (Kang & McAuley, 2018), have demonstrated promising results in sequential recommendation. Despite these advancements, there remains a gap in effectively leveraging IoT data to predict user equipment (UE) trajectories in mobile networks.

Current trajectory prediction methods face several key challenges:

• Reliance on GPS data: Many models depend heavily on GPS coordinates, which can be challenging to obtain or may be inaccurate in certain environments.

• Handling long sequences: Existing models often struggle to capture long-term dependencies and handle long sequences effectively.

• Computational efficiency: Traditional models are computationally expensive and not well-suited for parallel processing, leading to slow training times.

• Generalization and overfitting: Some models are prone to overfitting and do not generalize well to new data.

Contribution and novelty

To address these gaps, we propose a User-Embedded Temporal Attention Neural Network (UETANN) for predicting the next cell reached by UE in IoT environments. Our contributions are as follows:

• UE embedding: By incorporating UE embedding, we enable the model to learn the behavioral characteristics of users. This allows the model to predict locations based on similar user behaviors, effectively handling sparse sequence data and improving generalization capabilities.

• Optimized attention mechanism: We enhance the attention mechanism by eliminating redundant query matrices and attention masks. This reduction in parameters not only speeds up training but also enhances computational efficiency, addressing a gap that previous studies have not adequately addressed.

• Sliding window sampling: We utilize sliding window sampling to leverage more trajectory information. This approach enhances the model’s ability to capture complex patterns and long-term dependencies, which are often overlooked in traditional models.

• Negative sampling: By selecting nearby locations as negative samples, we help the model better distinguish between positive and negative samples, thereby increasing prediction accuracy. This technique fills a gap in the literature where previous methods have struggled to improve precision in dense and overlapping spatial distributions.

Our extensive empirical study shows that the recall of our algorithm reaches 0.5766, indicating the optimal result and high performance of our model. This performance is particularly significant in the context of IoT trajectory prediction, where accurate and timely predictions are crucial for optimizing network resources and enhancing user experience.

Existing approaches for trajectory prediction

Several recent studies have explored the use of deep learning and attention mechanisms for trajectory prediction and recommendation. For instance, Spatial temporal recurrent neural networks (STRNN) (Liu et al., 2016) uses time and space interval information to improve location recommendation precision (Wu et al., 2016), while Spatio-temporal gated network (STGN) (Zhao et al., 2022) enhances LSTM models for better performance. Attention based spatiotemporal LSTM (ATST-LSTM) (Huang et al., 2021) and spatial-temporal attention networks (STAN) (Luo, Liu & Liu, 2021) also leverage spatial-temporal information to improve model accuracy.

Recent works, such as the Augmented Intelligence of Things for Emergency Vehicle Secure Trajectory Prediction and Task Offloading (Wu et al., 2024), TRACE: Transformer-based Continuous Tracking Framework Using IoT and MCS (Mohammed, Singh & Otrok, 2024), and Trajectory Simplification and Adaptive Map Matching Algorithm for Electric Bicycle (Wang, Liu & Yu, 2023), highlight the importance of integrating IoT data and advanced machine learning techniques for real-time tracking and prediction. These studies provide valuable insights into the challenges and opportunities in the field.

Limitations of current methods

Despite their effectiveness in certain scenarios, existing methods have several limitations:

• Reliance on GPS data: Many models depend heavily on GPS coordinates, which can be challenging to obtain or may be inaccurate in certain environments.

• Handling long sequences: Existing models often struggle to capture long-term dependencies and handle long sequences effectively.

• Computational efficiency: Computing efficiency is an important consideration for IoT (Potluri et al., 2011; Feng et al., 2015), while traditional models are computationally expensive and not well-suited for parallel processing, leading to slow training times.

• Generalization and overfitting: Some models are prone to overfitting and do not generalize well to new data.

Comparison with existing methods

Our proposed model overcomes the limitations of existing methods in the following ways:

• Independence from GPS data: Our model does not rely on GPS data, making it suitable for scenarios where location coordinates are missing or inaccurate.

• Handling long sequences: By using sliding window sampling, our model leverages more sequence information, improving its ability to handle long sequences effectively.

• Reducing computational load: Optimizing the attention mechanism by removing redundant query matrices and attention masks reduces the number of parameters, speeding up the training process.

• Enhancing generalization: Sliding window sampling and negative sampling techniques help the model generalize better, reducing overfitting and improving prediction accuracy.

History trajectory

A UE trajectory is temporally ordered arrivals. Each arrival $a_k$ of the trajectory of UE $u_i$ is a tuple ( $u_i$, $c_k$, $t_k$), in which $c_k$ is the cell and $t_k$ is the timestamp. Each UE has a variable length trajectory $\mathrm{t}\mathrm{r}\mathrm{a}\left(u_i\right)=\left\{a_1,a_2,\dots ,a_{m_i}\right\}$, where $m_i$ is the total length of the $u_i$ trajectory. Every trajectory is transformed into several fixed length sequences $\mathrm{s}\mathrm{e}\mathrm{q}\left(u_i\right)=\left\{a_1,a_2,\dots ,a_n\right\}$, with n as the maximum length of a single $\mathrm{s}\mathrm{e}\mathrm{q}\left(u_i\right)$ we consider. We only consider trajectories whose length is not less than 5. If $m_i$ < n, zeros are padded for cells to the left until the trajectory length is n, and we pad timestamp with the first timestamp $t_{s_1}$ where $s_1$ is not a padded cell so that those time intervals will be zero after calculation as we can see in the “Time Intervals Embedding Layer” section. Samples of $u_i$ will be like this: from

(1) $$\left\{a_1,a_2,...,a_n\right\}\mathrm{t}\mathrm{o}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}{\mathrm{c}}_{\mathrm{n}+1},$$

$$\left\{a_2,a_3,...,a_{n+1}\right\}\mathrm{t}\mathrm{o}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}{\mathrm{c}}_{\mathrm{n}+2},\cdots$$

$$\left\{a_{m_i-n},a_{m_i-n+1},...,a_{m_i-1}\right\}\mathrm{t}\mathrm{o}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}{c}_{m_i}.$$

We mask off the padding arrivals during computation.

Mobility prediction

Given the UE trajectory $\left(a_1,a_2,\dots ,a_m\right)$, our target is to predict the next arriving cell c $\in$ $a_{m+1}$.

Multimodal embedding layer

The multimodal embedding layer is designed to encode the UE, cell, and time intervals into latent representations. This layer comprises three sub-layers: the UE trajectory embedding layer, the positional embedding layer, and the time intervals embedding layer.

Attention layer

The attention layer is responsible for learning the relationships between the arrivals within the UE trajectory. By focusing on relevant parts of the trajectory, the attention mechanism helps the model capture long-term dependencies and improve prediction accuracy. We remove the query matrix operation and the attention mask to reduce the number of parameters and speed up training. The attention mechanism updates the representation of each arrival by weighting the contributions of other arrivals in the trajectory.

Negative sampling

Negative sampling is used to transform the original prediction problem into a binary classification problem. For each positive sample (the actual next cell), we select several negative samples (nearby cells that are not the next cell). This approach helps the model better distinguish between positive and negative samples, thereby increasing prediction accuracy.

Recall that we convert UE trajectory to several fixed length sequences $\mathrm{s}\mathrm{e}\mathrm{q}\left(u\right)=\left\{a_1,a_2,...,a_n\right\}$, $a_i=\left(u,c_i,t_i\right)$. We define ${\mathit{o}}_n$ as the expected output given a n length UE sequence. Usually, we adopt negative sampling to increase score of positive cells and decrease score of negative cells. We use the six adjacent cells of ${\mathit{c}}_n$ as candidate cells, including ${\mathit{c}}_{n-1}$ and the next cell that UE actually arrives, sometimes the two will coincide, the next cell that UE actually arrives is a positive sample, and we randomly select one of the remaining five cells as a negative sample. Thus, we get expected positive cell ${\mathit{o}}_n$ and one negative cell ${\mathit{o}}_n^{\mathrm{\prime }}$ to obtain a set of pairwise samples $D=\left\{\left(\mathrm{s}\mathrm{e}\mathrm{q}\left(u\right),o,o^{\prime }\right)\right\}$.

Prediction layer

The prediction layer computes the score of candidate cells. Given the updated representations from the attention layer, the prediction layer calculates the likelihood of each candidate cell being the next destination. We use the six adjacent cells of the current cell as candidates due to spatial limitations. The scores are computed using a scoring function, and the cell with the highest score is selected as the predicted next cell.

We obtain the composite representation of arrivals, locations, and time intervals following stacked attention blocks. In the last block, we only compute $s_n$, as well as ${\alpha }_{nj}$, $g_{nj}$, $S_n$, regardless of previous n − 1 results. To distinguish from the previous expression, we rename the last output $S_n$ to ${\mathit{S}}_n^{\prime }$. To predict the next cell, we multiply the representation learned before and cell embedding to calculate UE’s preference score for cell i, as shown below:

(15) $$R_{i,n}={\mathit{S}}_n^{\mathrm{\prime }}{\mathit{C}}_i$$where ${\mathit{C}}_i$ $\in {\mathrm{R}}^d$ is the embedding of cell i and ${\mathit{S}}_n^{\prime }$ is the representation given the previous n arrivals (i.e., $a_{s_1}$, $a_{s_2}$, . . ., $a_{s_n}$) and their time intervals matrix ${\mathit{R}}^u$.

Cross-entropy loss

To optimize the model, we use cross-entropy loss. The loss function increases the score of positive samples and decreases the score of negative samples. This ensures that the model learns to accurately predict the next cell while maintaining robustness against incorrect predictions.

We use binary cross entropy as the loss function and a sigmoid function σ(x) = 1/(1 + exp(−x)) to make cell scores between zero and one:

(16) $$-\sum _D\left[\log \left(\sigma \left(R_{o_n,n}\right)\right)+\log \left(1-\sigma \left(R_{o_n^{\mathrm{\prime }},n}\right)\right)\right]+\lambda \parallel \mathrm{\Theta }{\parallel }_F^2$$where ${\parallel \cdot \parallel }_F$ stands for the Frobenius norm, λ is the regularization parameter, and $\mathrm{\Theta }$ is the set of embedding matrices.

The Adam (Kingma & Ba, 2017) optimizer is used to improve our suggested model. We use mini-batch SGD (Stochastic Gradient Descent) to increase training efficiency since the number of samples could be very big.

Detailed flow diagram

Figure 3 is a detailed flow diagram of the model architecture:

Explanation of the flow diagram:

• 1. Input layer: Raw data (UE ID, cell ID, and check-in time) is fed into the model.

• 2. Multimodal embedding layer:

  • UE trajectory embedding: Converts UE and cell IDs into dense vectors.

  • Positional embedding: Adds positional information to maintain sequence order.

  • Time intervals embedding: Encodes time intervals between arrivals.

• 3. Attention layer: Captures relationships between arrivals using a modified attention mechanism.

• 4. Negative sampling: Transforms the problem into a binary classification task by selecting negative samples.

• 5. Prediction layer:

  • Candidate cell scoring: Computes scores for candidate cells.

  • Prediction: Selects the cell with the highest score as the next cell.

• 6. Loss function: Uses cross-entropy loss to optimize the model.

Dataset

Data source

The experimental data are from the historical data detail record (DDR) generated by the IoT devices. We make experiments on two product forms, shared bicycle and goods locate. The time span is from June 20, 2022 to July 21, 2022. Figure 4 shows what UE trajectories may look like; some cells are frequently reached while others are not.

The above are private datasets, and we are also conducting experiments on public datasets, mainly using the following public datasets:

• MovieLens: The MovieLens dataset is a public dataset produced by the GroupLens project team. The MoveieLens dataset can be said to be one of the most classic datasets in the field of recommendation systems, which records when each user reviews what movies. We use the version MovieLens-1m, which contains over 1 million records (Harper & Konstan, 2015).

• Gowalla: A check-in dataset, widely used for location prediction and POI prediction, which is introduced in Cho, Myers & Leskovec (2011).

• Foursquare: Foursquare is a mobile service website based on user geographic location information, encouraging mobile users to share their current geographic location and other information with other people. The Foursquare dataset records this information, and is introduced in Sarwat et al. (2014).

• Amazon: The Amazon dataset records users’ evaluations of Amazon website products and is a classic dataset for recommendation systems. We mainly selected two types, “Beauty” and “Electronics”, which are introduced in Ni, Li & McAule (2019).

Measurement indicator

The performance of recommendations is assessed using Recall@k and NDCG@k, in order to confirm the viability of the model described in this research. In our scenario, Recall@k measures the ratio of the number of relevant results retrieved from the top K results to the number of all relevant results, and NDCG@k considers the ranking of recalled products, higher values are given to those with higher rankings. We remove the negative sampler module from the assessment process and directly remember the target from positive samples. The performance improves as the metrics increase.

Implementation details

We implemented the UETANN model using PyTorch, a widely-used deep learning framework. The detailed architecture and hyperparameters used in our experiments are provided below:

Performance on private data and ablation study

The relationship between training loss and recall on shared bicycle data is shown in Figs. 5 and 6. As we can see, the loss function drops rapidly at the beginning, and then slowed down, and gradually stabilized after 30 epochs. With the decline of the loss function, the evaluation metrics rise rapidly at the beginning, and then tend to be flat. Unlike the steady decline of the loss function, the evaluation metrics will fluctuate. This is because the loss function distinguishes one positive sample from one negative sample, while the evaluation has to distinguish one positive sample from five negative samples, so it is inevitable that the evaluation metrics will fluctuate slightly, but generally they rise.

After 40 epochs, we get confusion matrix and receiver operating characteristic (ROC) plots on training data as shown in Figs. 7 and 8, where confusion matrix is a tabular tool used to evaluate and visualize the performance of classification models, widely used in machine learning and statistical classification tasks. It presents the correspondence between the predicted results of a classifier and the actual categories in the form of a matrix, reflecting the classification performance of the model on each category, including the number of correctly classified and misclassified instances. Both the number of positive and negative samples are 4,060, and the area under the curve (AUC) 0.84 infers that our model shows a good effect. Table 2 shows the performance of our algorithm.

If UE embedding is not included in the model, the model effect is not much less than the original. We guess the reason is that the number of cells is too much more than that of UEs.

Without using sliding window sampling, recall@1 of shared bicycle is 0.5469, less than 0.5766 in Table 2. This is because the model uses sliding windows to adopt more trajectory points, and learned more information.

If we choose negative samples not from the nearest five cells, but randomly from tens of thousands of cells, then ultimately recall@1 of shared cycle is 0.5378, less than 0.5766. This is because selecting negative samples from the nearest five cells for training aligns with the goal of evaluating model performance on the validation and testing sets, making the model more capable of distinguishing positive and negative samples. Therefore, it is important to choose negative samples reasonably.

We also conducted experiments on retaining query matrix, and found that recall did not change much.

Comparison of different methods on public datasets

To demonstrate the performance of our algorithm, we have also conducted extensive experiments on public datasets, comparing our algorithm with multiple algorithms. Note that our algorithm focuses on data without latitude and longitude, so the algorithms we compared also do not use latitude and longitude coordinates. We mainly compared with the following nine methods.

• TCFRA (Song et al., 2016): This method combines temporal features and collaborative filtering.

• Bayesian personalized ranking (BPR) (Rendle et al., 2009): This method uses Bayesian personalized ranking for POI recommendation.

• TransRec (He, Kang & McAuley, 2017): This is a transaction-based recommendation algorithm.

• GRU4Rec+ (Hidasi & Karatzoglou, 2018): This algorithm models user behavior sequences using RNN networks with improved loss functions and sampling methods.

• MARank (Yu et al., 2019): The article proposes a multi-level attention ranking model that unifies individual level and union level into the inference model of user intention from multiple perspectives.

• Factorizing personalized Markov chains (FPMC) (Rendle, Freudenthaler & Schmidt-Thieme, 2010): This approach integrates matrix factorization with a first-order Markov chain to model both long-term user preferences and short-term dynamics.

• Caser (Tang & Wang, 2018): It maps the recent sequence of items into a specific time interval within the latent space, enabling the capture of higher-order Markov chains that consider the L most recent items.

• Beyond self-attention for sequential recommendation (BSASRec) (Shin et al., 2024): It proposes an attentive inductive bias method that surpasses the self-attention mechanism to improve the performance and effectiveness of sequential recommendation systems.

• SASRec-SCE (Mezentsev et al., 2024): This article proposes a scalable cross-entropy (SCE) loss function that efficiently approximates full cross-entropy loss for large catalogs, reducing memory usage by up to 100 times without sacrificing recommendation accuracy.

The experimental results are shown in Table 3. Apart from one metric where it is slightly behind BSASRec, the proposed method outperforms all other algorithms across all other metrics.

Influence of hyper-parameters

We find the best hyper-parameters by hyperopt which is a tool to adjust parameters through Bayesian optimization and of fast speed. With step 10, we increase the dimension of embedding d from 10 to 70. Figures 9 and 10 demonstrates that d = 60 is the optimal parameter. We experiment a series of number of sequence length n = (20, 30, 40, 50, 60, 70). Figures 11 and 12 manifests that n = 60 is the best number for sequence length.

Visualization

As mentioned earlier, due to the lack of cell longitude and latitude information, we cannot visualize the movement trajectory. But after cell embedding, we get the cell representation in high-dimensional space, and reduce the dimension from 60 to 2 through T-distributed stochastic neighbour embedding (TSNE) algorithm to get the cell representation in two-dimensional space. Adjacent cells are closer in the plane, as shown in Fig. 13.

We select some trajectory points of two UEs to make two trajectory diagrams, to visualize the prediction process and results of our algorithm. Figures 14 and 15, the historical trajectory is from Cell 44 to Cell 60, Cell 61 is the next cell to arrive, and the green dots are the other five candidate cells, which infer our predicted results are perfectly coincident with the real position of trajectory. The results from the trajectory diagram display that our algorithm performs well and has high prediction accuracy.

Figures 16 and 17 shows the average attention the models pay to each trajectory point, the padding cells are removed when calculating the average attention weights. Both the two datasets indicate that more attention has been paid to the recent cells.

Discussion of experimental results

To provide a clearer demonstration of how UETANN outperforms previous baselines, we conduct a detailed analysis of the experimental results. Our method integrates UE embedding, cell embedding, and visit time interval information, alongside sliding window sampling and an optimized attention mechanism. These components collectively enhance the model’s ability to predict the next cell in a UE’s trajectory.