Multi-angle information aggregation for inductive temporal graph embedding

Temporal graph embedding

In the field of graph embedding, researchers commonly categorize graphs into two main types: static graphs and dynamic graphs. A static graph refers to a graph where neither the topological structure nor the node attributes change over time. In the early stages of graph embedding, researchers primarily focused on the topological structure of graphs. They obtained the adjacency matrix of the graphs and utilized techniques such as random walk or matrix decomposition (Ou et al., 2016) to learn node embeddings. Random walk means that, in a simple one-dimensional random walk, an entity starts at a specific point and at each time step, it moves either left or right with equal probability. The path taken by the entity forms a random walk. Matrix decomposition, also known as matrix factorization, is the process of breaking down a matrix into simpler, more manageable parts. This is commonly done to simplify calculations, extract meaningful information, or reduce the complexity of a problem.

For instance, DeepWalk performs a random walk procedure over the graph and then applies the Skip-Gram (Mikolov et al., 2013a) model to learn node embeddings (Perozzi, Al-Rfou’ & Skiena, 2014). LINE learns node embeddings by considering first-order and second-order proximity (Tang et al., 2015), while node2vec introduces a biased random walk procedure to balance breadth-first and depth-first search strategies (Grover & Leskovec, 2016).

In contrast to static graphs, dynamic graphs encompass the changes that occur in a graph over time. Compared with static graphs, dynamic graphs can better reflect the real changes in the real world. Whether in academic citation scenarios, social scenarios, or business scenarios, the establishment of edges is never completed at the same time, and there may be a long time interval between them. If you cannot grasp this time information, you may make wrong judgments. Capturing the temporal evolution of graph structures allows researchers to obtain more effective embeddings. In the early stages of dynamic graph research, graphs were divided into several states based on timestamps, with each state representing a static snapshot of the dynamic graph (Liang et al., 2023). Current work often employs graph neural network (GNN) frameworks to learn node embeddings within each static snapshot at different timestamps, leveraging recurrent neural network (RNN) frameworks to capture temporal changes in node embeddings (Xu et al., 2019). For example, DySAT computes node embeddings through joint self-attention across structural neighborhoods and temporal dynamics (Sankar et al., 2020), while EvolveGCN captures dynamic graph changes by using an RNN to evolve the parameters of the graph convolutional network (GCN) (Pareja et al., 2020). These methods can utilize time information without changing the classic static graph technology, and also make researchers think more deeply about whether they must stick to the adjacency matrix training model. In this case, the temporal graph came into being.

Recognizing that static snapshots may not accurately represent graph changes, researchers have shifted their focus toward learning node embeddings in temporal graphs characterized by chronological interactive events (Liu & Liu, 2021; Fan, Liu & Liu, 2022). CTDNE incorporates temporal information into node embeddings through biased or unbiased random walk procedures (Nguyen et al., 2018), while HTNE employs the Hawkes process to capture the influence of historical neighbors on the current node to obtain node embeddings (Zuo et al., 2018). JODIE models the future trajectory of node embeddings and introduces a novel projection operator that learns to estimate node embeddings at any future time (Kumar, Zhang & Leskovec, 2018). AGLI (Liu, Wu & Liu, 2022) focuses on the global influence and local influence aggregation. TGC (Liu et al., 2024b) is the first work to focus on deep temporal graph clustering. The information comparison of these methods is shown in Table 1.

Transductive learning and inductive learning

Graph embedding approaches can be categorized into two types based on the training pattern: transductive learning and inductive learning.

Transductive learning aims to generate fixed node embeddings by directly optimizing the final state of the graph. Many existing approaches for node embedding fall under this category. However, transductive learning has a drawback in dynamic graphs: when the graph undergoes changes, these approaches require retraining the entire graph to obtain updated node embeddings, which can be computationally expensive. Consequently, transductive learning is not well-suited for generating new node embeddings in dynamic graphs. As mentioned above, assume that for a graph dataset with 10,000 nodes and 100,000 interactions, it may take 20 epochs, or about half an hour, to complete training. When a new node is added, it will take half an hour to retrain each time. In the real world, the size of this graph dataset may increase by a hundred times, and the frequency of new nodes joining may be reduced to every hour. In this case, there will be a situation where new data is added before the previous round of retraining is completed, which is unrealistic.

In contrast, inductive learning focuses on learning a model that can generate node embeddings at any given time. This approach does not produce fixed embeddings but instead emphasizes the ability to calculate node embeddings directly when new nodes are added. The model leverages the node’s features and neighborhood information to compute its embedding. For instance, GraphSAGE learns a function that generates embeddings by sampling and aggregating features from a node’s local neighborhood (Hamilton, Ying & Leskovec, 2017). DeepGL is a deep hierarchical framework designed for large graphs, capable of discovering both node and edge features (Rossi, Zhou & Ahmed, 2018). DyREP introduces a two-time scale deep temporal point process model to capture the interleaved dynamics of observed processes (Trivedi et al., 2019).

Based on this classification, our proposed method MIAN falls into the category of inductive learning in dynamic temporal graphs. Temporal graphs offer an accurate embedding of graph changes in real-world datasets, while inductive learning enables flexible capturing of these changes. Therefore, MIAN is better suited for generating effective node embeddings in dynamic temporal graphs. MIAN considers information fusion from multiple angles, focusing on the local neighborhood angle, time angle, and global environment angle, and fuses them together through the GRU module to generate node embeddings with richer information content. In the following sections, we will provide detailed insights into the MIAN method.

Problem definition

As discussed previously, we can learn node embeddings in the graph inductively. According to the interaction between nodes, we can formally define the temporal graph as follows.

Definition 1: Temporal graph. When two nodes establish an interaction within the graph, the interaction is accompanied by a clearly defined timestamp. Accordingly, the temporal graph can be formally defined as G = (V, E, T), where V represents the set of nodes, E⊆V × V denotes the set of edges, and T represents the set of interaction timestamps. For each edge e = (u, v) connecting node u and node v, there exists at least one corresponding interaction instance denoted as T_u,v = (u, v, t₁), (u, v, t₂), …, (u, v, t_n), where t_i represents an individual timestamp associated with the interaction between nodes u and v.

In a temporal graph, a node engages in multiple interactions with other nodes, and these interactions can be arranged in chronological order based on their timestamps. When two nodes interact with each other, we refer to them as neighbors. The historical neighbor sequence of a node can be defined as follows.

Definition 2: Historical neighbor sequence.

For a given node u, its historical neighbor sequence H_u can be obtained, where H_u = (v₁, t₁), (v₂, t₂), …, (v_n, t_n). Each tuple in this sequence represents an event, indicating that node v_i interacts with node u at time t_i.

Motivated by the work presented in Liu, Wu & Liu (2022), our objective within the context of the aforementioned temporal graph is to develop a function capable of capturing multi-angle information for the purpose of inductively updating node embeddings. The following sections will provide detailed explanations of our method.

Overall framework

As shown in Fig. 1, we consider the potential information in the temporal graph embedding from three different perspectives. Furthermore, since neighborhood-angle information and temporal-angle information are naturally close, we combine the information from these two angle to generate neighborhood with temporal information embeddings, and also generate environment information embeddings. These information embeddings are fused with the node embedding itself in the improved GRU module to generate the final node embedding.

Neighborhood angle information

Neighborhood information plays a crucial role in graph neural networks, and many traditional graph learning methods employ message propagation mechanisms to aggregate neighborhood information for nodes. This involves combining the features of different neighboring nodes with varying weights as part of the node’s own characteristics. This approach effectively enables the learning of information-rich node embeddings while maintaining the independence of each node’s embedding to a certain extent.

In this process, determining how to assign weights to information from different neighboring nodes becomes an area of significant exploration. Following an interaction between node u and v, node v will influence the future interactions of node u with other nodes, and reciprocally, node u will influence the future interactions of node v with other nodes. This influence can be interpreted as different weighting methods for aggregating the characteristics of different neighboring nodes. Here, we adopt two approaches to construct such weights.

Temporal angle information

In the realm of temporal graph learning, time information holds significant importance. Drawing inspiration from the work outlined in Hawkes (1971), the Hawkes process is employed to model discrete sequential events. This process assumes that past events have an impact on the occurrence of future events. According to this framework, the probability of future events is influenced by historical events, with the influence diminishing over time.

In the context of our problem, we posit that a node’s historical neighbors play a role in shaping its future interactions. Moreover, the closer the interaction in terms of time, the greater the influence it exerts on future interactions. Consequently, we can calculate the time weight k_u,i for neighbor i on node u using the following equation. (5)${\displaystyle k_{u,i}=exp\left(-{\delta }_u^t|t_c-t_i|\right)}$

where t_i is the timestamp when neighbor i interacts with u, t_c is the current time, ${\delta }_u^t$ is a learnable weight parameter that regulates the neighbor nodes’ time weight on u.

The temporal weights calculated above will be further incorporated into the embedding of the neighborhood angles as part of the weight. In this way, the temporal with neighborhood information embedding can be calcluated as follows. (6)${\displaystyle z_u^{NT}=\sum _{i\in H_u}{\theta }_{u,i}{z}_i^{t_{n-1}},{\theta }_{u,i}=exp\left(p_{u,i}\cdot {\omega }_{u,i}\cdot k_{u,i}\right)}$

Indeed, incorporating the time weight into Eq. (4) enhances the embedding of node u by considering both the neighborhood weight and the temporal aspect. In a temporal graph, the time information encapsulates the interactions between a node and its neighbors. Consequently, although these two types of information are derived from different perspectives, they are inherently interconnected and can be effectively combined to improve the overall embedding of the node.

Environment angle information

The interactions among nodes within a graph have the potential to alter the graph’s structure and properties, subsequently influencing the global environment. Conversely, the global environment can also impact the interactions between nodes. As illustrated in the accompanying figure, when a node u initially becomes part of a graph, it may exhibit limited sensitivity to changes occurring in the global environment. However, once the cumulative affinity of node u surpasses a certain threshold, it becomes attuned to global environment changes and can readily capture the latest updates within the graph.

This mechanism highlights the notion that nodes gradually become more responsive to the evolving global environment as they accumulate stronger relationships and affinities with their neighbors. It enables nodes to adapt and incorporate the most recent changes, allowing for a more comprehensive understanding of the graph dynamics.

Aggregator function

In order to consolidate the various angle information, we propose an enhanced version of the GRU module to serve as an aggregator function. The GRU module, introduced by Cho et al. (2014), is capable of capturing temporal patterns in sequential data by controlling the degree of aggregation for different information and determining the proportion of historical information to retain. In particular, We extend GRU to devise an aggregator function that combines multiple-angle information embeddings with the node embeddings from the previous timestamp to generate the node embeddings for the next timestamp. The aggregator function employed is defined as follows. (12)${\displaystyle U{G}_u=\sigma \left(W_{UG}\left[z_u^{t_{n-1}}\oplus z_u^{NT}\oplus z_u^E\right]+b_{UG}\right)}$ (13)${\displaystyle TN{G}_u^{t_n}=\sigma \left(W_{TNG}\left[z_u^{t_{n-1}}\oplus z_u^{NT}\oplus z_u^E\right]+b_{TNG}\right)}$ (14)${\displaystyle E{G}_u^{t_n}=\sigma \left(W_{EG}\left[z_u^{t_{n-1}}\oplus z_u^{NT}\oplus z_u^E\right]+b_{EG}\right)}$ (15)${\displaystyle {\tilde{z}}_u^{t_n}=tanh\left(W_z\left[z_u^{t_{n-1}}\oplus \left(TN{G}_u^{t_n}\odot z_u^{NT}\right)\oplus \left(E{G}_u^{t_n}\odot z_u^E\right)\right]+b_z\right)}$ (16)${\displaystyle z_u^{t_n}=\left(1-U{G}_u\right)\odot z_u^{t_{n-1}}+U{G}_u\odot {\tilde{z}}_u^{t_n}}$

Here σ is the sigmoid function, ⊕ denotes concatenation operator, ⊙ denotes element-wise multiplication. $z_u^{NT}$, $z_u^E$ and $z_u^{t_n}$ are temporal with neighborhood information embedding, environment information embedding and node u’s embedding at time t_n respectively. W_UG, W_TNG, W_EG, W_z ∈ ℝ^d×3d, b_UG, b_TNG, b_EG, b_z ∈ ℝ^d are learnable parameters, UG, TNG, EG ∈ ℝ^d are called update gate, local reset gate, and global reset gate respectively.

In this study, we introduce a modification to the GRU by dividing the reset gate into two separate gates: the temporal with neighborhood reset gate (TNG) and the environment reset gate (EG). The TNG gate and EG gate are utilized to control the degree of information retention for the two types of embeddings, respectively. By combining the node embedding from the previous timestamp and the two reserved information embeddings, we generate a new hidden state $\tilde{z}{u}^{tn}$ for the next timestamp.

Furthermore, we employ the UG gate to control the degree of historical information retention. Using the node embedding $z_u^{t_{n-1}}$ from the previous timestamp and the new hidden state $\tilde{z}{u}^{tn}$ for the next timestamp, we derive the node embedding $z_u^{t_n}$ for the next timestamp. This recursive process allows us to calculate node embeddings iteratively.

Loss function

In order to learn effective node embeddings in a fully unsupervised setting, we employ a graph-based loss function on the node embedding $z_u^{t_n}$ and optimize it using the Adam method proposed by Kingma & Ba (2015). The graph-based loss function aims to encourage similar embeddings for neighboring nodes, while ensuring that embeddings of dissimilar nodes are significantly different. To measure the similarity between two embeddings, we utilize the negative squared Euclidean distance. The loss function is defined as follows: (17)${\displaystyle logL=\sum _{u\in V}\sum _{v\in H_u}\left[log\sigma \left(-{∥z_u^{t_n}-z_v^{t_n}∥}^2\right)-Q\cdot E_{v_n\sim P_n\left(v\right)}log\sigma \left(-{∥z_u^{t_n}-z_{v_n}^{t_n}∥}^2\right)\right].}$

To address the significant computational burden associated with the loss function, we employ negative sampling (Mikolov et al., 2013b) to optimize the loss. Negative sampling involves sampling negative examples from a distribution P_n(v), where Q denotes the number of negative samples. In our approach, we sample negative nodes that have not appeared in the historical neighbor sequence of node u. This helps alleviate the computational cost while still allowing us to train the model effectively. The algorithm procedure is given in Algorithm 1 .

Note that in the link prediction task, there are some other loss functions, which are briefly introduced here for reference. Binary Cross Entropy Loss (BCELoss) is a common loss function used for binary classification tasks, where the output is a probability score between 0 and 1. Hinge loss is commonly used for binary classification tasks as well, especially in support vector machines (SVMs). It penalizes predictions that are on the wrong side of the decision boundary. Bayesian Personalized Ranking Loss (BPR Loss) is often used in collaborative filtering tasks, including link prediction in recommendation systems. It optimizes the ranking of positive interactions over negative interactions.

 
_______________________ 
Algorithm 1 MAIN procedure______________________________________________________________ 
Require: Temporal Graph G. 
Ensure: Node embeddings for downstream tasks. 
  1:  Initialize model parameters and data loading; 
  2:  Fetch data by batch; 
  3:  repeat 
 4:     for each epoch do 
 5:         for each batch do 
 6:            Calculate neighborhood angle information in Section 3.3; 
  7:            Calculate temporal angle information in Section 3.4; 
  8:            Calculate environment angle information in Section 3.5;; 
  9:            Aggregate the multi-angle information via GRU; 
10:            Optimize the loss function and update the parameters; 
11:         end for 
12:     end for 
13:  until Convergence__________________________________________________________________________    

Datasets

We report the statistical information of the following real-world graph datasets in Table 2. The datasets we selected are several public datasets commonly used in the temporal graph community, which are often used by researchers for experimental analysis. These datasets cover different scenarios such as academic, social, and business, and can more comprehensively analyze the real performance and versatility of the model.

DBLP (Zuo et al., 2018): This dataset represents a co-authorship graph in the field of Computer Science. It contains over 10,000 nodes, and for our study, we extracted 10 research fields from the DBLP website.

BitCoin (Kumar et al., 2016; Kumar et al., 2018): The BitCoin dataset is derived from a bitcoin trading platform called Alpha. Users on the platform rate other members on a scale of -10 (total distrust) to +10 (total trust) with steps of 1. We categorized the ratings into seven distinct categories based on every three consecutive scores.

ML1M (Li, Wang & McAuley, 2020): ML1M is a widely used movie dataset employed in various machine learning tasks. Each movie in this dataset is associated with a score assigned by users, and we choose the most frequently rated score as the label for each movie. The scores range from 1 to 5, allowing us to divide the movies into five categories.

Amazon (Ni, Li & McAuley, 2019): The Amazon dataset consists of magazine subscription data. Similar to the ML1M dataset, each magazine is assigned a score based on user ratings. We select the most frequently assigned score as the label for each magazine, and the magazines are divided into five categories accordingly.

Baselines

We compare MIAN with multiple state-of-the-art baselines:

CTDNE (Nguyen et al., 2018) gives rise to methodologies for acquiring time-respecting embeddings from networks with continuous-time dynamics.

HTNE (Zuo et al., 2018) employs the Hawkes process to capture the influence of historical neighbors on the current node, thereby obtaining node embeddings.

JODIE (Kumar, Zhang & Leskovec, 2019) represents a coupled recurrent neural network model that learns the embedding trajectories of users and items.

TGN (Rossi et al., 2020) constitutes a versatile and efficient framework for deep learning on dynamic graphs represented as sequences of timed events.

TREND (Wen & Fang, 2022) introduces a framework that incorporates both event and node dynamics, thereby enabling a more precise modeling of events within a Hawkes process-based graph neural network.

OTGNet (Feng et al., 2022) offers a comprehensive and principled learning approach for open temporal graphs, aiming to address the aforementioned challenges.

The code of our proposed method MIAN can be find in GitHub: https://github.com/doublefish-han/MIAN.

Experimental settings

The experimental results were obtained using a desktop computer equipped with an Intel Core i7-6800K CPU, an NVIDIA GeForce RTX 3090 GPU, 64 GB of RAM, and the PyTorch deep learning framework.

All comparison methods were utilized with their default parameters, without any modifications. For our proposed MIAN method, we set the activation threshold to 0.9, the node embedding dimension size to 128, the number of training epochs to 5/10/20, and the batch size and learning rate to 128 and 0.001, respectively.

We compare MIAN with other methods on several real-world datasets under the node classiciation task. We also conduct parameter sensitivity study, ablation study, and convergence analysis to verify the effectiveness of the proposed method.

Node classification performance

In the node classification experiment, we conduct the performance comparison between the proposed MIAN algorithm and several state-of-the-art temporal graph learning methods. The results, as reported in Table 3, demonstrate that MIAN outperformed all other methods on all datasets, providing evidence for the effectiveness of our proposed method.

Note that each result was subjected to five repetitions to determine its corresponding margin of error, revealing the relative stability of our method. Furthermore, it is worth noting that the performance improvement of our method varies across different datasets. This observation suggests that the model needs to consider varying weights of information from different perspectives depending on the distribution of the data. To delve deeper into this phenomenon, we conducted additional study on parameter sensitivity and performed ablation study on the MIAN method.

Parameter sensitivity study

In this part, we report the effectiveness of different activate threshold values on all datasets. As mentioned above, the activate threshold controls whether a node can compare the global environment information, thus further influence the final node embeddings.

Here we set the activate threshold as 0.10, 0.25, 0.5, 0.75, 0.90, respectively. Then we training MIAN model on these different parameter settings. In this process, all other hyper-parameters are set to default values and remain unchanged. As shown in Fig. 2, we can find that when the activate threshold comes 0.90, the performance usually become the best performance. In addition, in different temporal graph datsets, the fluctuation range of the effect caused by the change of threshold is not the same.

Indeed, for varying data distributions across different datasets, it is crucial to adaptively tune the hyper-parameter values in our model to align with the specific characteristics of each dataset. Additionally, it is worth highlighting that irrespective of how the threshold value may change, the performance fluctuation of MIAN remains within a narrow range. This observation underscores the robustness of our model, as it demonstrates the model’s ability to withstand variations in hyper-parameters without significant impact on its effectiveness.

Hyperparameters are values that need to be set manually. It is often difficult to find completely accurate values in practical application scenarios because the distribution and scale of real data are different, which may affect the selection of hyperparameter values. In this case, by selecting a small-scale public dataset of the same neighborhood (for example, both in the academic field) for preliminary parameter adjustment and partially extracting and adjusting parameters on internal data, researchers can often choose more reliable parameter values. The hyperparameters here are not limited to the content we discussed, but also include common hyperparameters such as embedding dimension size, training batch size, and number of epochs.

Ablation study

We also focus on the effectiveness of different angle information on the proposed MIAN method. In this part, we conduct ablation study to observe it.

In particular, we construct several variants of MIAN. Among them, the baseline model we used is called “BASE”, which does not make any changes. For information from different angles, we record them as “BASE+N”, “BASE+T”, and “BASE+E”, respectively, and the final complete model is “MIAN”.

As shown in Fig. 3, we report the ablation results on different datasets. From the figure, we can find that information from different angles improves the model performance to different extents, among which information from the neighborhood angle can bring the greatest improvement. This is consistent with our general understanding that in the field of graph embedding, information propagation brought by neighbors can greatly enhance the information contained in node embedding. Regardless of the angle of information, it can improve the effect of the model, and the final complete MIAN model has the best performance. In addition, on different datasets, due to different data distributions, the improvement brought by information from different angles is different. This is also consistent with what we said above.

Note that different perspectives are of different importance, which is also well understood in real-world data. Neighborhood information is the most important, followed by time information, and then environment information. Because in static graph learning, even without time information, a good prediction rate can be achieved because we know the objects these nodes interact with. Time information is a supplement to the details. Furthermore, if the environment does not change significantly, then the impact of the global environment is not significant. If the environment fluctuates drastically, then it is worth considering the global influence to a greater extent.

Convergence analysis

Here, we report the convergence of the model on different datasets in Fig. 4. It can be seen that our model training requires fewer epochs, that is, the loss value drops rapidly in the first five epochs, basically close to the convergence range, and then continuously adjusts in multiple epochs of training until convergence. This means that if a small loss of accuracy is acceptable, our model can be limited to a small number of training epochs, so that training can be completed quickly. Compared with other methods, the training speed is faster and the accuracy loss is lower.

In addition, we would like to point out that the change of loss function does not completely mean the change of effect. Sometimes the final adjustment of loss value does not bring about a significant improvement in performance. More specifically, the initial loss values on different datasets are different, so the thresholds of final convergence should also be different. Therefore, simply based on the change of loss function, we can only verify the speed of model convergence, but the final effect may fall on an earlier epoch. Therefore, we choose five epochs for BitCoin and ML1M, 10 epochs for DBLP, and 20 epochs for Amazon.