Social-aware trajectory prediction using goal-directed attention networks with egocentric vision

Bird’s-eye view and pedestrian-focused trajectory prediction models

Early trajectory prediction works primarily employed bird’s-eye view imagery to model pedestrian behavior. Alahi et al. (2016) introduced Social-LSTM, which uses recurrent neural networks (RNN) with a social pooling layer to capture interpersonal interactions. Dendorfer, Osep & Leal-Taixé (2020) proposed Goal-GAN, which decomposes prediction into goal estimation followed by trajectory refinement. Zhang & Li (2022) introduced the multimodal trajectory prediction transformer (MTPT) and Huang et al. (2019) introduced STGAT to fuse spatial and temporal interactions via GAT mechanisms. The availability of egocentric pedestrian datasets such as JAAD (Rasouli, Kotseruba & Tsotsos, 2017) and PIE (Rasouli et al., 2019) further enabled researchers to analyze pedestrian dynamics from a moving vehicle’s perspective. Subsequent frameworks like BiPed (Rasouli, Rohani & Luo, 2021) extended these approaches by jointly predicting pedestrian trajectories and actions, while methods such as the multi-information CNN by Wang et al. (2021) integrated historical trajectories and contextual cues to improve prediction accuracy.

To address inherent uncertainties in trajectory prediction, several multimodal frameworks have been developed (Li et al., 2024b; Ruan et al., 2024). Approaches such as ABC+ (Halawa, Hellwich & Bideau, 2022), SGNet (Wang et al., 2022a), and BiTraP (Yao et al., 2021) leverage CVAE to generate diverse trajectory hypotheses and quantify uncertainty. Complementary methods include the non-parametric deep ensemble approach by Li et al. (2024a). Huynh & Alaghband (2021) proposed GPRAR to reconstruct human poses and actions under noisy conditions, while Liu et al. (2024) introduced a two-stage method for modeling interpretable social interactions, and Lv et al. (2023) developed SSAGCN using social soft attention to simultaneously capture pedestrian interactions and environmental context. These studies collectively advance multimodal prediction but remain largely confined to homogeneous, pedestrian-centric scenarios.

Recent works have focused on refining trajectory prediction from an egocentric perspective by employing advanced feature extraction and goal estimation techniques. Dong et al. (2024) introduced TSNet to mitigate negative pedestrian character information, and Hu et al. (2025) developed PMTPN for probabilistic forecasting of vulnerable road user trajectories using multimodal inputs. Feng et al. (2024) employed a transformer-based forward generation model, while Astuti et al. (2024) presented CITraNet, which uses a Transformer and extreme value theory integration to predict trajectory and crossing intentions. Complementary to these, Chen et al. (2023) proposed a stacking model for real-time trajectory prediction of vulnerable traffic participants, Lin, Lin & Hung (2023) focused on predicting potentially dangerous pedestrian behavior with an Attention-LSTM architecture, Rasouli & Kotseruba (2023) introduced PedFormer for cross-modal pedestrian behavior prediction, and Sharma, Dhiman & Indu (2023) developed a visual–motion–interaction-guided framework for pedestrian intention prediction. Although these approaches improve egocentric predictions for pedestrians and bicyclists, they do not fully address heterogeneous traffic environments that include vehicles and motorcyclists, thereby motivating the need for more comprehensive models.

Vision-based multi stacked neural networks trajectory prediction models

Different from most studies that rely on a single encoder–decoder architecture for trajectory prediction, the present study leverages a vision-based multi-stacked neural network framework that integrates several specific modules to learn from sequential sensor data. The literature in this area can be broadly categorized into the following five components:

• 1.

  Graph-based models: Socially aware encoder architectures employing graph convolutional networks (GCN) (Huynh & Alaghband, 2021; Lv et al., 2023; Wang et al., 2022b) and graph neural networks (GNN) (Li et al., 2022a) have been widely used in edge graph trajectory prediction. Recently, GAT (Veličković et al., 2018) has emerged as a superior alternative, as it can adaptively learn more comprehensive graph representations than conventional GCN or GNN. For example, GraphTCN (Wang, Cai & Tan, 2021) utilizes a CNN-based spatial–temporal graph framework to capture edge features with salient spatial information, and STGAT (Huang et al., 2019) assigns importance to neighboring nodes to generate spatial interactions. In light of these developments, our framework incorporates a GAT network to implicitly assign spatial interaction weights within the encoder module.

• 2.

  LSTM-based models: Classic LSTM networks address the vanishing gradient problem encountered in RNNs. Social-LSTM (Alahi et al., 2016) aggregates hidden states of pedestrians using social pooling, and Gupta et al. (2018) combine LSTM cells with generative adversarial (GAN) networks. Moreover, LSTM networks have proven effective in encoding temporal information by preserving long-term dependencies in complex sequential sensor data (Chen et al., 2023; Sharma, Dhiman & Indu, 2023; Li et al., 2020). Our approach leverages LSTM networks to robustly encode the temporal dynamics present in the data.

• 3.

  Goal-based models: Recent research has incorporated goal networks to condition trajectory predictions. For instance, Goal-GAN (Dendorfer, Osep & Leal-Taixé, 2020) investigates multimodal conditioning routes for estimating final positions, ExpertTraj (Zhao & Wildes, 2021) generates predictions conditioned on goals derived from an expert repository, and GTP (Tran, Le & Tran, 2021) employs a dual-channel neural network to integrate goal information into future trajectory estimation. Additionally, BiTraP (Yao et al., 2021) and SGNET (Wang et al., 2022a) estimate multimodal goals. Motivated by these approaches, our framework includes a GDF module to generate successive goal estimation.

• 4.

  CVAE-based models: CVAE-based models have been extensively used for multimodal trajectory prediction, particularly in capturing individual driving styles and uncertainty in human behavior. ABC+ (Halawa, Hellwich & Bideau, 2022) employs an action-based contrastive learning loss, while AgentFormer (Yuan et al., 2021) infers agents’ latent intent to model social influence. CSR (Zhou et al., 2023b) enhances pedestrian prediction through cascaded sequential estimation, whereas DACG (Zhou et al., 2023a) refines human intention modeling by incorporating time-dependent social interactions. Importantly, CVAEs can condition trajectory generation on driver-specific attributes such as reaction times and maneuver tendencies, making them well-suited for personalized trajectory forecasting (Liu et al., 2024). Unlike graph-based networks, which primarily encode spatial and temporal dependencies, CVAEs enable diverse trajectory hypotheses that better capture individual motion variations under uncertainty (Chen et al., 2023). Recognizing these advantages, our study integrates a CVAE module to generate multimodal trajectories across homogenous and heterogeneous road users while considering driver-specific characteristics.

• 5.

  Attention-Based Models: Recent work has demonstrated that attention mechanisms enhance trajectory prediction by effectively mixing and regulating information. Lv et al. (2023) employed soft attention with GCN, and Li, Wang & Zuo (2023) integrated MHA and FFN into encoder and social interaction modules. Li et al. (2024b) further combined historical motion data with environmental context. These developments motivate our inclusion of MHA and feed-forward networks in both the decoder and the goal-directed forecaster (GDF) modules.

This study extends these advancements by integrating graph-based, LSTM-based, goal-directed forecaster, CVAE-based, and attention-based modules within a unified vision-based multi-stacked neural network framework. This comprehensive framework explicitly models socially aware interaction correlations and multimodal trajectory uncertainties for both homogeneous and heterogeneous road users.

Multimodal input information sequences

The proposed model processes multimodal input sequences using a feature extractor, as illustrated in Fig. 1. The detection and tracking system assign labels to all road users after recording traffic scenarios using the dashboard camera.

• Road user categories: Five road user types are considered, such as pedestrians, bikers, cyclists, trucks, and cars.

• Bounding box representation: Each road user’s trajectory is defined by its bounding box coordinates, which include top-left $\{x_{tl},y_{tl}\}$ and bottom-right $\{x_{br},y_{br}\}$ corners.

• Speed estimation: The speed of each road user is incorporated to model interaction weights and influence levels.

• Distance measurement: The distance between the ego-vehicle and each road user.

Since this study focuses on egocentric vision, odometry data from IMU and OBD sensors captures the ego-vehicle’s motion state over time. The accelerometer records longitudinal acceleration along the x-axis for forward movement $(m/s^2)$, while the gyroscope measures yaw velocity $(rad/s)$ alongInt the y-axis to track turning behavior. To ensure temporal synchronization, multimodal input streams are aligned using timestamps specific to each dataset. TITAN synchronizes camera and IMU sensor readings at 10 Hz, while PIE and JAAD synchronize camera and OBD sensor data at 30 FPS. Since different modalities operate at different frequencies, sequential frame alignment is performed to match sensor readings with their corresponding visual frames, ensuring consistency across input streams. These multimodal inputs are then concatenated into a unified sequence:

(1) $$X^t=\left\{b^t,s^t,d^t,o^t\right\}$$where $b^t$ represents road users’ classes $c^t$ and bounding box coordinate. $s^t$ is the speed of each road user as $s^t=\{x_{tl,br}^{t_c}-x_{tl,br}^{t_c-1},y_{tl,br}^{t_c}-y_{tl,br}^{t_c-1}\}$ at current frame $t_c$. $d^t$ denotes the distance between ego-vehicle $o$ and road user $n$ in a scene, which are calculated into $d^t=\{(x_n-x_o),(y_n-y_o)\}$. $o_t$ is odometry information as the state of ego-vehicle over the time.

Historical and future time sequences

At current frame $t$, the historical information of each target road user $n$ is given as:

(2) $$H_n^t=\left\{X^{t_{obs}},...,X^t\right\}$$where $H_n^t$ is the observed historical sequence, $X^t$ represents the concatenated input, and $t$ is the maximum historical sequence length. The objective is to predict future trajectories as:

(3) $$Y_n^t=\left\{b^{t+1},b^{t+2},...,b^{t+T_f}\right\}$$where $Y_n^t$ represents the future bounding box trajectory for each of road user $n$ over a prediction horizon $T_f$ frames.

Feature extractor

Each input sequence is embedded into a 512-dimensional feature vector using a fully connected (FC) network, ensuring a structured representation for downstream processing:

(4) $$e_n^{t_{obs}}={\mathrm{\psi }}_{\mathrm{e}\mathrm{m}\mathrm{b}}(X;W_{emb})$$where ${\mathrm{\psi }}_{\mathrm{e}\mathrm{m}\mathrm{b}}$ is the embedding sub-layer with learned weight matrix $W_{\mathrm{e}\mathrm{m}\mathrm{b}}$. Following this, the proposed SGANet processes these embeddings to learn multimodal interactions and predict future trajectories up to the maximum prediction horizon $T_f$.

Spatial interaction encoder module

Recent studies emphasize the importance of modeling interactions between road users and their neighbors (Huynh & Alaghband, 2021; Li et al., 2022a; Veličković et al., 2018; Wang et al., 2021). Our method models each road user as a node in a fully connected graph, where edges represent interaction dependencies. Figure 2, based on the TITAN dataset (Malla, Dariush & Choi, 2020), illustrates the complete GAT network for social interactions, while Fig. 3 provides a focused view of a single GAT layer, specifically highlighting interactions centered on node h1. This distinction clarifies the broader interaction framework versus node-specific attention modeling, offering a comprehensive perspective on the spatial interaction encoder.

Our model employs GAT to quantify social interactions by dynamically assigning influence weights to different road user classes. The attention mechanism prioritizes interactions based on proximity, speed, trajectory direction, and class-specific behavioral tendencies, allowing the model to weigh certain road users more heavily in trajectory prediction. For instance, faster-moving vehicles receive higher attention scores when approaching a pedestrian’s trajectory, while stationary objects receive minimal weighting. Additionally, interactions between pedestrians and nearby vehicles are weighted more heavily than interactions between two distant vehicles, ensuring that socially relevant behaviors are emphasized.

Each node in the GAT layer is represented by the input feature $h=\{{\stackrel{\rightharpoonup }{h}}_1,{\stackrel{\rightharpoonup }{h}}_2,...,{\stackrel{\rightharpoonup }{h}}_N\}$, where ${\stackrel{\rightharpoonup }{h}}_N\in {\mathbb{R}}^F$ and $F$ is the feature dimension of each node. Given an observed frame, the GAT layer receives input with $h^{t_{obs}}=\{b^{t_{obs}},s^{t_{obs}}\}$. The coefficients between node $i$ and its neighboring node $j$ can be computed by:

(5) $${\alpha }_{ij}^{t_{obs}}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}(\mathrm{E}\mathrm{L}\mathrm{U}(a^T[W{h}_i^{t_{obs}}\parallel W{h}_j^{t_{obs}}]))}{{\sum }_{k\in N(i)}\mathrm{e}\mathrm{x}\mathrm{p}(\mathrm{E}\mathrm{L}\mathrm{U}(a^T[W{h}_i^{t_{obs}}\parallel W{h}_k^{t_{obs}}]))}}$$where ${\alpha }_{ij}^{t_{obs}}$ is the attention coefficient of node $j$ to $i$ at the observed time sequences $t_{obs}$, $N_i$ represents the neighbors of node $i$ on the graph, the normalization is using softmax with ELU activation function, $a\in {\mathbb{R}}^F$ is weight vector of single FFN layer, $W$ is the learned weight matrix of inputted information sequence embedding $h^{t_{obs}}\in e_n^{t_{obs}}$ and a shared linear transformation which is applied to each node, and $\parallel$ is the concatenation operation.

Once the attention coefficients are computed, the updated representation of node $i$ in the GAT layer is obtained as:

(6) $$A_i^{t_{obs}}=\sigma \left({\sum }_{j\in N(i)}{\alpha }_{ij}^{t_{obs}}W{h}_j^{t_{obs}}\right)$$where σ represents nonlinear activation functions. Equations (5) and (6) describe a single GAT layer to aggregate socially aware interaction features. $A_i^{t_{obs}}$ is the result of the aggregated hidden state and contains the socially-aware correlation interaction from the target road user $i$.

Temporal encoder module

As shown in Fig. 4, the proposed encoder module utilizes an LSTM network to capture temporal dependencies, where solid and dashed arrows indicate the current time $t_c$ and the observation time sequences $t_{obs}$, respectively. After extracting the object feature $e_n^{t_{obs}}$ and spatial interaction encoding $A_n^{t_{obs}}$ for each road user $n$, these representations are concatenated along with the aggregated goal information $Y_{goal}^{t_{obs}}$. The goal feature encoder, which encodes the bounding box ground truth at the maximum prediction horizon $T_f$, is derived from the GDF module and serves as the goal feature at the current time step.

Given a road user $n$ in a scene, the object feature embedding $e_n^{t_{obs}}$, spatial interaction encoding from the GAT layer $A_n^{t_{obs}}$, and goal feature $Y_{goal}^{t_{obs}}$ are concatenated and fed into an LSTM cell. This process is formulated as:

(7) $$m_{enc}^{t_{obs}}=\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(A_n^{t_{obs}},e_n^{t_{obs}},Y_{goal}^{t_{obs}}),m_n^{t_c-1};W_m)$$where $m_{enc}^{t_{obs}}$ represents the hidden state of the LSTM cell at the current time step $t_c$ up to the observation state $t_{obs}$ with its learned weight matrix $W_m$. The initial hidden state $Y_{goal}^{t_{obs}}$ is set to a zero vector. These parameters are shared across all road users in the scene, and the encoder output $m_{enc}^{t_{obs}}$ is subsequently fed into the GDF and CVAE modules.

Goal-directed forecaster module

The GDF module generates coarse-to-fine goals for trajectory prediction and refines future goals at each time step. Figure 5 illustrates its overall structure, where solid and dashed arrows indicate current $t_c$ and predicted $t_{pre}$ time sequences. Given a predicted goal bounding box location $(x_{tl}^{T_f},y_{tl}^{T_f},x_{br}^{T_f},y_{br}^{T_f})$ at the maximum future time $T_f$, with the initial goal set as a zero-filled vector $\mathrm{t}=0$. The GDF module predicts coarse goals, which are passed to the encoder and CVAE modules, generating goal loss. This study uses twenty noisy goals to $m_{enc}^t$ at each time step t to evaluate their impact on prediction accuracy before being processed by the CVAE module.

A set of observed goal locations $Y_{goal}^{t_{obs}}$ is concatenated with the input to enhance the temporal encoder $m_{enc}^{t_{obs}}$, enabling it to learn discriminative representations across timsequences. At each future step $t_c+t_{pre}$, the CVAE module utilizes a subset of consecutive goals $Y_{goal}^{t_c+t_{pre}}$ as coarse guidance for trajectory prediction. The goal regressor $Y_{reg}^{T_f}$ compresses multiple goals into a single representation, optimizing the goal loss function.

To improve prediction accuracy, a goal aggregator using soft attention is introduced to weigh successive goals before passing them to the encoder and CVAE modules. The input data $m_{enc}^{t_{obs}}$ is reshaped to match the goal embedding size using a linear transformation followed by ELU activation:

(8) $$m_g^{t_c+1}=\mathrm{E}\mathrm{L}\mathrm{U}(W_g{m}_{enc}^{t_{obs}}+b_g)$$where $m_g^{t_c+1}$ represents hidden goals at time $t_c+1$ and $b$ is the bias term. The sequence $m_g^{t_c+1}$ is then processed by an LSTM cell, defined as:

(9) $$m_g^{t_c+t_{pre}}={\sigma }_g(\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}(m_g^{t_c+1},m_g^{t_c+t_{pre}};W_g)).$$

The MHA network is applied to refine consecutive hidden goal inputs $m_g^{t_c+t_{pre}}$, where $m_g^{t_c+t_{pre}}=\{m_g^{t_c+1},...,m_g^{t_c+t_{pre}},...,m_g^{t_c+T_f}\}$. Given queries $Q$, keys $K$, and values $V$ from the hidden goal network $m_g^{t_c+t_{pre}}$, each attention head computes:

(10) $$\mathrm{A}\mathrm{t}\mathrm{t}(Q_h,K_h,V_h)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(Q{K}^T/\sqrt{d_k})V$$where $d_k$ is the scaling factor, and each head has its own weight matrices such as $Q_h=Q{W}_{Q_h}$, $K_h=K{W}_{K_h}$, $V_h=V{W}_{V_h}$. Then, the MHA network’s output $\mathrm{A}\mathrm{t}\mathrm{t}(Q_h,K_h,V_h)$ is passed through a soft attention mechanism $\alpha$, which assigns attention weights to $l^{th}$ predicted goal scores that can be written as:

(11) $${\alpha }_l={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}(e^T)}{{\sum }_{n=1}^N\mathrm{e}\mathrm{x}\mathrm{p}(e^T)}}$$where $e^T=a^T(W_a\mathrm{A}\mathrm{t}\mathrm{t}(Q_h,K_h,V_h)+b_a)$. The computed attention weights ${\alpha }_l$ are then summed with the normalized weights, producing observed goal location $Y_{goal}^{t_{obs}}$ and predicted goal location $Y_{goal}^{t_c+t_{pre}}$. These are passed to the encoder and CVAE modules, as follows:

(12) $$Y_{goal}^{t_{obs}}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(\mathrm{E}\mathrm{L}\mathrm{U}(W_{enc}{\alpha }_l+b_{enc}))$$

(13) $$Y_{goal}^{t_c+t_{pre}}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(\mathrm{E}\mathrm{L}\mathrm{U}(W_{dec}{\alpha }_l+b_{dec})).$$

Finally, to regularize the GDF module, we regress $Y_{reg}^{T_f}$ to infer a single coarse goal representation, defined as:

(14) $$Y_{reg}^{T_f}={\varphi }_{reg}(\mathrm{A}\mathrm{t}\mathrm{t}(Q_h,K_h,{\mathrm{V}}_h);W_{reg}).$$

Conditional variational autoencoder module

This study adopts CVAE for multimodal trajectory prediction. Figure 6 illustrates the CVAE module structure, where the red dashed line indicates operations using ground truth during training, the solid blue line represents common flows in both training and testing, and the solid green line is exclusive to testing. The CVAE module consists of three sub-networks:

• 1.

  The conditional prior network $p_{\theta }(Z|m_{enc}^{t_{obs}})$

• 2.

  The recognition network $q_{\varphi }(Z|m_{enc}^{t_{obs}},E)$

• 3.

  The trajectory generation network $p_{\omega }(m_{Latent}^{t_c+t_{pre}}|m_{enc}^{t_{obs}},Y_{goal}^{t_c+t_{pre}},Z)$

where $\theta$, $\varphi$, and $\omega$ represent their respective network parameters, $Y_{goal}^{t_c+t_{pre}}$ is GDF module output (from Eq. (9)) and $m_{enc}^{t_c-t_{obs}}$ denotes the encoder module output (from Eq. (4)). All networks are implemented using FC layers.

CVAE module during training process

The recognition network $q_{\varphi }(Z|m_{enc}^{t_{obs}},E)$ captures dependencies between the observed trajectory state $m_{enc}^{t_{obs}}$ and the ground truth trajectory $E$. The ground truth trajectory $E$ is processed by a bidirectional LSTM (Bi-LSTM) to generate the hidden state $m_E^{t_c+t_{pre}}$, which computed as:

(15) $$m_E^{t_c+t_{pre}}=\mathrm{B}\mathrm{i}-\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}(E,m_E^{t_c+1};W_E)$$where $E$ is the target ground truth trajectory $\{e_n^{t_c+1};...;e_n^{t_c+t_{pre}};...;e_n^{t_f}\}$ at future time sequence $t_c+t_{pre}$ for each road user $n$, and $e_n$ consists of coordinate position $b^t$, distance $d^t$, speed $s^t$, and odometry ${\mathrm{o}}^{\mathrm{t}}$. The recognition network does not assume a predefined distribution for $E$ but implicitly learns it through the distribution $Z_q\in Z$ and $Z_q$ calculation is as follows:

(16) $$Z_q=q_{\varphi }(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(m_{enc}^{t_{obs}},m_E^{t_c+t_{pre}});W_q)$$where ${\mu }_{Z_q}$ and ${\sigma }_{Z_q}$ represent the predicted distribution mean and covariance to captures the dependencies between the observed trajectory $m_{enc}^{t_{obs}}$ and the ground truth $m_E^{t_c+t_{pre}}$.

CVAE module during testing process

During inference, the prior network $p_{\theta }(Z|m_{enc}^{t_{obs}})$ samples multiple latent variables $Z$ to generate diverse trajectory predictions. This enables the model to capture different driving styles, reaction times, and trajectory patterns, ensuring the predicted trajectories reflect individual variations in behavior. By leveraging multimodal predictions, CVAE accounts for uncertainties in driver decision-making, such as varying acceleration tendencies or reaction delays to obstacles.

Moreover, the prior network $p_{\theta }(Z|m_{enc}^{t_{obs}})$ operates without knowledge of the ground truth $E$. Instead, it predicts the distribution mean ${\mu }_{Z_p}$ and covariance ${\sigma }_{Z_p}$ using a subset of $Z$, which consists of the predicted goal location $Y_{goal}^{t_c+t_{pre}}$ and the encoder input $m_{enc}^{t_{obs}}$. Multiple trajectory samples are drawn from $N({\mu }_{Z_p},{\sigma }_{Z_p})$ to generate the latent variable $Z_p$, formulated as:

(17) $$Z_p=p_{\theta }(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(m_{enc}^{t_{obs}},Y_{goal}^{t_c+t_{pre}});W_p).$$

To generate multimodal trajectory predictions $Y_n^{t_c+t_{pre}}$, the trajectory generation network $m_{Latent}^{t_c+t_{pre}}$ utilizes the latent space by concatenating samples from $N({\mu }_{Z_p},{\sigma }_{Z_p})$ and $N({\mu }_{Z_q},{\sigma }_{Z_q})$ along with $m_{enc}^{t_{obs}}$ and $Y_{goal}^{t_c+t_{pre}}$, expressed as:

(18) $$m_{Latent}^{t_c+t_{pre}}=p_{\omega }(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(m_{enc}^{t_{obs}},Y_{goal}^{t_c+t_{pre}},Z);W_p).$$

CVAE module uses Kullback-Leibler divergence loss

To address the absence of the ground truth target $E$ during testing, the Kullback-Leibler (KL) divergence loss $(D_{KL})$ between $N({\mu }_{Z_p},{\sigma }_{Z_p})$ and $N({\mu }_{Z_q},{\sigma }_{Z_q})$ is minimized. This ensures that the prior network $p_{\theta }(Z_p|m_{enc}^{t_{obs}})$ implicitly learns the dependency between the observed trajectory $m_{enc}^{t_{obs}}$ and the future trajectory $E$. The KL loss optimization is formulated as:

(19) $$D_{KL}[q_{\varphi }(Z_q|m_{enc}^{t_{obs}},E)\parallel p_{\varphi }(Z_p|m_{enc}^{t_{obs}})]=\sum _{Z\in K}{q}_{\varphi }(Z_q|m_{enc}^{t_{obs}},E)\mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{p_{\theta }(Z_p|m_{enc}^{t_{obs}})}{q_{\varphi }(Z_q|m_{enc}^{t_{obs}},E)}}.$$

Decoder module

The decoder module predicts the trajectory distribution from the CVAE latent space $m_{Latent}^{t_c+t_{pre}}$. Figure 7 illustrates the overall decoder structure, which consists of a MHA, FFN, and FC networks. The solid and dashed arrows represent the current time step $t_c$ and prediction time sequences $t_c+t_{pre}$, respectively. To decode the predicted trajectory $Y$, where $Y=[Y_n^{t_c+1};...;Y_n^{t_c+t_{pre}};...Y_n^{t_c+T_f}]$ represents the predicted future bounding box coordinate location for each road user. The decoder module leverages MHA to capture diverse relationships within the input data. Based on Eq. (10), for each attention head $h$, the latent space $m_{Latent}^{t_c+t_{pre}}$ from the CVAE module is linearly transformed into $Q_h,K_h,V_h$ and the attention score is computed using the scaled dot product of queries $Q_h$ and keys $K_h$.

The FFN then processes the output of MHA, transforming it into a new hidden state $m_{ffn}^{t_c+1}$ for the next time step $t_c+1$ using layer normalization, formulated as:

(20) $$m_{ffn}^{t_c+t_{pre}}=\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(m_{ffn}^{t_c+1}+\mathrm{A}\mathrm{t}\mathrm{t}(Q_h,K_h,V_h)).$$

Finally, the trajectory regressor, a single FC layer, takes $m_{ffn}^{t_c+t_{pre}}$ as input to compute the trajectory prediction $Y$ of each road user $n$ at the next future time step $t_c+t_{pre}$, expressed as:

(21) $$Y_n^{t_c+t_{pre}}={\mathrm{\psi }}_{\mathrm{d}\mathrm{e}\mathrm{c}}(Y_n^{t_c+1};W_{dec}).$$

Loss functions

This study adopts the best-of-many (BoM) approaches to minimize the distance between the best predicted trajectory and the ground truth trajectory, where $\hat{Y}=[{\hat{Y}}_n^{t_c+1};...;{\hat{Y}}_n^{t_c+t_{pre}};...;{\hat{Y}}_n^{t_c+T_f}]$ represents the ground truth at future time sequences $t_c+t_{pre}$ and $T_f$ is the maximum future time. Multimodal trajectory prediction enhances model accuracy and diversity by capturing data variation. Root mean squared error (RMSE) is used as the loss function for the encoder module $L_{Traj}$.

To ensure accurate goal network trajectory predictions, the goal prediction loss $L_{Goal}$ is optimized using RMSE between the predicted goal $Y_{reg}^{T_f}$ and the ground truth goal ${\hat{Y}}^{T_f}$. The KL loss $(D_{KL})$ is applied to the CVAE module $L_{CVAE}$ to reduce differences between the recognition and prior networks. The total loss function for each of training iteration is given by:

(22) $$L_{total}=L_{Traj}+L_{Goal}+L_{CVAE}.$$

Each loss function is formulated as follows:

(23) $$L_{Traj}=\underset{\mathrm{\forall }k\in K}{min}\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}(Y,\hat{Y}),L_{Goal}=\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}(Y_{reg}^{T_f},{\hat{Y}}^{T_f})$$where $K$ is the total number of trajectory samples.

For the CVAE loss function $L_{CVAE}$, the stochastic latent variable $Z$ is sampled from a Gaussian distribution, and the CVAE objective is to maximize its variational lower bound:

(24) $$\underset{\theta ,\varphi ,\omega }{\mathrm{m}\mathrm{a}\mathrm{x}}{\mathbb{E}}_{q_{\varphi }(Z_q|m_{enc}^{t_c-t_{obs}},E)}[\mathrm{l}\mathrm{o}\mathrm{g}{p}_{\omega }(m_{Latent}^{t_c+t_{pre}}|m_{enc}^{t_{obs}},Y_{goal}^{t_c+t_{pre}},Z)]-{\mathrm{D}}_{KL}[q_{\varphi }(Z_q|m_{enc}^{t_c-t_{obs}},E)\parallel p_{\theta }(Z_p|m_{enc}^{t_c-t_{obs}})]$$where the first term maximizes the log-likelihood of the predicted target distribution, and the KL term minimizes the difference between the recognition and conditional prior networks. Finally, the total loss function $L_{total}$ is progressively minimized through backpropagation, adjusting parameters and updating weights iteratively until convergence.

Datasets

To evaluate the effectiveness of the proposed SGANet model, this study utilizes both homogeneous and heterogeneous datasets. JAAD (Rasouli, Kotseruba & Tsotsos, 2017) and PIE (Rasouli et al., 2019) are selected due to their focus on pedestrian behavior in egocentric traffic scenarios, making them ideal for analyzing human-vehicle interactions. Both datasets contain video sequences recorded at 30 FPS from dashboard-mounted cameras, providing high-resolution pedestrian trajectory data. In contrast, the TITAN dataset (Malla, Dariush & Choi, 2020) is chosen for its diverse road user types, including pedestrians, cyclists, motorcyclists, and vehicles, making it suitable for evaluating multimodal trajectory prediction in heterogeneous traffic environments. TITAN is captured using a GoPro Hero 7 with an IMU sensor at 10 Hz (10 FPS), providing synchronized ego-motion data essential for modeling motion dynamics. For JAAD and PIE datasets are split into training, validation, and testing sets, following the 50% overlap approach from Rasouli, Rohani & Luo (2021); For TITAN dataset, we follow the same dataset split as same as TITAN model (Malla, Dariush & Choi, 2020).

Implementation details

The proposed SGANet model was trained on an Ubuntu 20.04 system with an NVIDIA GeForce RTX 3090 GPU for 100 epochs, using the Adam optimizer with a learning rate of 0.0001, a batch size of 128, and a ReduceLROnPlateau scheduler that reduces the learning rate by 0.1 after five epochs without validation loss improvement (minimum threshold: 1e−10). The encoder module processes multimodal inputs, including label class, bounding box coordinates, speed, distance, and IMU/OBD sensor readings, embedding them into 512-dimensional FC layers. The GDF module refines future goal predictions with 128-dimensional goal embeddings, enhanced by MHA with eight attention heads. The CVAE module utilizes a 32-dimensional latent space, with a Gaussian prior (nu = 0.0, sigma = 1.5), generating 20 trajectory samples per forward pass (K = 20). A Bi-LSTM (hidden size = 256) processes future goal locations, while LSTMCells (hidden size = 512) handle sequential dependencies in the trajectory encoder.

To capture socially-aware correlation interaction, SGANet models each road user as a node in a fully connected graph, using GATs to encode interactions. A 1-dimensional social embedding with a single attention head prioritizes neighboring influences, while MHA networks process social interactions. The trajectory encoder integrates bounding box data, goal features, and social interactions, using an LSTMCells (hidden size = 512). The CVAE module consists of a prior network, recognition network, and trajectory generation network. The prior network predicts distribution mean and covariance using goal locations and encoder input features, while the recognition network refines the latent space via a Bi-LSTM (hidden size = 256). The decoder employs MHA (four attention heads) to generate 4D bounding box coordinates for each road user at future timesteps.

Evaluations metrics

The performance of the SGANet model is evaluated using multimodal trajectory prediction, implementing the BoM approaches from K predicted samples. To assess future object localization, we use average displacement error (ADE), final displacement error (FDE), and final intersection over union (FIOU) metrics. ADE and FDE measure the positional error of the bounding box center, while FIOU evaluates the accuracy of the bounding box size and overlap. FIOU is included in the evaluation because ADE and FDE only measure localization error without considering the bounding box dimensions, which are crucial for understanding the spatial accuracy of bounding box predictions. For the TITAN dataset, the model observes 1 s of past data to predict a 2-s future trajectory (Malla, Dariush & Choi, 2020). For the JAAD and PIE datasets, it observes 0.5 s to forecast 1 s ahead (Rasouli, Rohani & Luo, 2021).

Models comparison

We evaluate SGANet on the TITAN dataset (Malla, Dariush & Choi, 2020) alongside several baseline models, such as Social-LSTM (Alahi et al., 2016), Social-GAN (Gupta et al., 2018), Const-Vel (Schöller et al., 2020), TITAN model (Malla, Dariush & Choi, 2020), GPRAR (Huynh & Alaghband, 2021, BiTraP (Yao et al., 2021), ELMA (Li et al., 2022b), CITraNet (Astuti et al., 2024). Then, the proposed SGANet on JAAD (Rasouli, Kotseruba & Tsotsos, 2017) and PIE (Rasouli et al., 2019) alongside with several baseline models that only use unimodal trajectory prediction (K = 1), such as PIE model (Rasouli et al., 2019), BiPed (Rasouli, Rohani & Luo, 2021), PedFormer (Rasouli & Kotseruba, 2023), Attention-LSTM (Lin, Lin & Hung, 2023), CITraNet (Astuti et al., 2024).

Ablation studies

Table 1 presents the ablation study results on the TITAN dataset (Malla, Dariush & Choi, 2020), evaluating different module integrations using ADE, FDE, and FIOU metrics.

Convergence results

The model’s convergence during training and validation is measured using the total error function defined in Eq. (22). The proposed model was trained for 100 epochs, with loss values stabilizing between 0.3 and 0.4 after approximately 30 epochs, as shown in Fig. 8. This indicates that the model effectively learns trajectory patterns while incorporating multimodal predictions and socially aware interaction correlations, ensuring robust trajectory forecasting across multiple road user classes.

Computational cost

Table 2 presents the computational cost of the proposed SGANet, evaluating parameters (PARAMS in M), floating point operations (FLOPs in G), and inference time (seconds). The study compares the computational efficiency of the temporal encoder module with GRU and LSTM networks. While GRU requires fewer gates and parameters, LSTM’s more complex architecture increases inference time. Although SGANet is computationally intensive due to its vision-based multi-stacked neural network design, ablation studies indicate that while GRU offers faster inference, it is less effective and adaptable for trajectory prediction.

Socially-aware correlation interaction visualization result

To evaluate our model’s ability to capture implicit socially-aware interaction correlations, we visualize the spatial attention weights learned during training (Fig. 9) using color-coded sequences. Our SGANet employs a GAT to quantify social interactions by dynamically assigning influence weights to different road user classes based on spatial proximity, speed, trajectory direction, and behavioral tendencies. This enables differential weighting of road users, ensuring that the most socially relevant interactions are emphasized in trajectory prediction. For instance, pedestrians at crosswalks or near the ego-vehicle exert a higher influence on trajectory adjustments compared to distant vehicles, while cyclists and motorcyclists require adaptive weighting due to their dynamic maneuverability.

The visualization highlights how the model prioritizes interactions dynamically, with larger circles representing higher influence levels. The largest circle indicates the road user most critical to the ego-vehicle’s next predicted movement, reflecting the adaptive nature of SGANet’s graph attention mechanism. By assigning higher weights to fast-approaching vehicles and nearby pedestrians while minimizing the impact of stationary objects, SGANet effectively models complex multi-agent interactions. This visualization confirms the model’s ability to intelligently infer and adjust interaction dependencies, leading to more accurate, socially-informed trajectory predictions in dynamic traffic environments.

Quantitative comparison results

Qualitative results

Heterogeneous TITAN dataset visualizations

Figure 10 illustrates the observed (cyan), ground truth (red), and predicted (green) trajectories in various driving scenarios. This study presents four qualitative trajectory predictions: (a) and (b) depict multi-target road user predictions, while (c) and (d) focus on a single target’s predicted destination and bounding box. The multimodal trajectory distribution is visualized through 20 sampled predictions (e, f) and KDE-based density estimations (g, h). These results indicate that SGANet enhances prediction robustness, helping autonomous vehicles anticipate sudden changes and accurately estimate road users’ movement directions.

Figure 11 further examines trajectory predictions over extended time horizons (2 to 7 s). As shown, longer prediction intervals (represented by dot markers) result in greater deviation from the ground truth, especially beyond 6 s. In first-person view (FPV) datasets, extended predictions lead to smaller road users being filtered out, as seen in Fig. 11, where only one road user remains visible after 70 frames. These findings highlight the challenge of long-term trajectory forecasting due to accumulated uncertainty and occlusion effects.

Homogeneous JAAD and PIE datasets visualization

Figure 12 illustrates the visualization results on the JAAD (A), (B), (C) and PIE (D), (E), (F) datasets, highlighting the multi-target pedestrian trajectory predictions generated by the proposed SGANet. The visualization demonstrates SGANet’s ability to model pedestrian movement across diverse urban scenarios, accurately capturing spatial interactions, pedestrian behaviors, and trajectory uncertainties. By leveraging socially-aware correlations, goal-directed forecaster, and multimodal trajectory prediction, the model can anticipate the robustness in predicting pedestrian trajectories under varied lighting conditions, occlusions, and urban complexities. Notably, the predicted trajectories align closely with the ground truth, validating the model’s capacity to generate reliable, socially compliant, and context-aware predictions. These results suggest that SGANet is effective in improving safety-critical decision-making for autonomous vehicles by reducing uncertainty in pedestrian intent estimation and future trajectory forecasting.

Extreme and complex scenario visualization

Figure 13 presents a sequence of successive frames showcasing multimodal trajectory predictions using Gaussian KDE in diverse and dense urban scenarios. The visualization highlights the model’s ability to generalize across varying traffic conditions, including complex interactions between pedestrians, cyclists, and vehicles. The predicted trajectories capture multiple possible trajectory patterns, effectively handling uncertainties in unpredictable pedestrian behavior and high-density urban traffic. In complex scenarios, such as crowded intersections or sudden pedestrian crossings, the model maintains robust performance by leveraging socially-aware interaction correlations and goal-directed forecasting. Additionally, the CVAE module enables the model to capture variations in driving styles and pedestrian movement unpredictability by generating diverse trajectory samples based on individual behavioral tendencies. This ensures that different reaction times, acceleration patterns, or abrupt pedestrian crossings are better accounted for in the trajectory prediction process. These qualitative results complement the reported ADE, FDE, and FIOU metrics, reinforcing the model’s adaptability and accuracy in extreme traffic situations. These dynamic visualizations further demonstrate the model’s ability to maintain accurate trajectory predictions over successive frames, adapting to diverse traffic conditions and complex interactions in real-world urban scenarios.