SAMGAT: structure-aware multilevel graph attention networks for automatic rumor detection

Weighted multi-head attention

To understand and represent different types of responsive relationships, we introduce multi-head attention (Vaswani et al., 2017) in Eq. (5) by averaging the outputs from K attention heads: (5)${\displaystyle h_i^{l+1}=ReLU\left(\frac{1}{K}\sum _{k=1}^K\sum _{j\in {\mathcal{N}}_i}{\alpha }_{ij}^{lk}{W}_k^l{h}_{jk}^l\right)}$

where $h_i^{l+1}$ signifies the latent representations of the post ω_i in the (l + 1)-th layer. ${\alpha }_{ij}^{lk}$ indicates the normalized attention weight determined by the k-th head in the l-th layer, and $W_k^l$ refers to the learnable parameters associated with the linear transformation at that layer. The resulting embedding for the l + 1 layer is obtained through an averaging process.

After going through L − 1 layers of GAT, we propose weighted multi-head attention to manage the contributions of different relationships. Equation (5) is further enhanced by Eq. (6) via weighting different relationships. This extension allows our model to differentiate and prioritize diverse perspectives and attitudes in user comments, building upon the foundation laid by the GAT. The computation of the final output in the L-th layer is calculated using a newly introduced weight vector c. (6)${\displaystyle h_i^L=ReLU\left(\sum _{k=1}^K{c}_k\sum _{j\in {\mathcal{N}}_i}{\alpha }_{ij}^{L-1,k}{W}_k^{L-1}{h}_j^{L-1}\right)}$

$h_i^L$ represents the enhanced node representation of ω_i, which encodes the weighted contributions of different relationships. We utilize mean pooling to summarize the whole event. (7)${\displaystyle \bar{s}=meanpooling\left(H^L\right)}$

Here, $\bar{s}$ refers to the representation of the entire graph obtained through mean-pooling.

DynamicGAT

Building upon the GAT, we introduce DynamicGAT, which includes two variations of the attention mechanism, namely GAT Original (GO) and dot-product (DP). These forms of attention aim to further refine our model, allowing it to better understand and represent data: (8)${\displaystyle e_{ij,\text{GO}}^l=LeakyReLU\left({\overrightarrow{a}}^{\top }\left[{\mathit{W}}^l{\mathit{h}}_i^l\parallel {\mathit{W}}^l{\mathit{h}}_j^l\right]\right)=LeakyReLU\left({\overrightarrow{a}}_1^{\top }{\mathit{W}}^l{\mathit{h}}_i^l+{\overrightarrow{a}}_2^{\top }{\mathit{W}}^l{\mathit{h}}_j^l\right)=LeakyReLU\left({\mathit{W}}_1^l{\mathit{h}}_i^l+{\mathit{W}}_2^l{\mathit{h}}_j^l\right)}$

where ${\overrightarrow{a}}_1\parallel {\overrightarrow{a}}_2=\overrightarrow{a}$, and ${\overrightarrow{a}}_1^{\top }{\mathit{W}}^l$ and ${\overrightarrow{a}}_2^{\top }{\mathit{W}}^l$ can be collapsed into two single linear transformations ${\mathit{W}}_1^l$ and ${\mathit{W}}_2^l$, respectively, and (9)${\displaystyle e_{ij,\text{DP}}^l={\left({\mathit{W}}^l{\mathit{h}}_i^l\right)}^{\top }\cdot \left({\mathit{W}}^l{\mathit{h}}_j^l\right)}$

Building on the enhanced capabilities of the GATv2 model (denoted as GO_v2) (Brody, Alon & Yahav, 2022), our approach incorporates a shared parameter weight W and instigates nonlinearity through the placement of the LeakyReLU function before ${\overrightarrow{a}}^T$. This modification enables GATv2 to achieve a universal approximation attention function, thereby endowing it with greater expressive power than its predecessor, GAT. The original GAT model operates under a static attention mechanism, where the ranking of attention scores among neighboring nodes remains constant across different central nodes. This means that regardless of whether the central node represents support or opposition, GAT assigns the highest attention to the same neighbor node—presumably the one with the highest support. Ideally, the model should aggregate neighbors that align with the central node’s stance. In contrast, GATv2′s formulation takes into account the relationship between the central node and its neighbors, allowing for a dynamic attention mechanism that can vary the attention scores based on the context provided by the central node: (10)${\displaystyle e_{ij,{\text{GO}}_{v2}}^l={\overrightarrow{a}}^{\top }LeakyReLU\left({\mathit{W}}^l{\mathit{h}}_i^l+{\mathit{W}}^l{\mathit{h}}_j^l\right)}$

Moreover, the ultimate attention mechanism of DynamicGAT, denoted as MX, is obtained by blending the GO and DP types. Specifically, MX is calculated by element-wise multiplication of GO_v2 and DP attention values followed by a sigmoid activation function σ. In the field of natural language inference (NLI), DP attention mechanisms have been widely employed to assess the compatibility between two sentences (Hu et al., 2020). Since DP attention, combined with the sigmoid function, represents the degree of agreement, it can subtly weaken the influence of nodes that contradict the view while implicitly emphasizing those that share a common perspective. (11)${\displaystyle \text{DynamicGAT}:e_{ij,{\text{MX}}_{v2}}=e_{ij,{\text{GO}}_{v2}}\cdot \sigma \left(e_{ij,\text{DP}}\right)}$

Constrained local attention

Experiments have confirmed that attention dropout is helpful for GAT training on small graphs. Fang et al. (2023) proves that integrating non-biased random dropping strategies into graph neural networks equates to adding an extra term for regularization, which leads to a more robust model. Taking inspiration from the dropkey technique utilized in the Swin-transformer (Li et al., 2023), we prefer to use dropkey rather than traditional dropout when computing the final attention scores. We specifically introduce a novel dropout-before-softmax scheme where the Key is set as the dropout unit. Importantly, we generate a unique masked key map for each central node instead of sharing a single map across all query vectors. This scheme can regularize attention weights and keep them as a probability distribution simultaneously. (12)${\displaystyle d_{ij}=\left\{\begin{array}{c}\hfill \text{0 with probability}1-d\hfill \\ \hfill -\infty \text{with probability}d\hfill \end{array}\right.}$ (13)${\displaystyle {\alpha }_{ij}=\frac{exp\left(d_{ij}+e_{ij}\right)}{\sum _{k\in \mathcal{N}\left(i\right)}exp\left(d_{ik}+e_{ik}\right)}.}$

Furthermore, in order to control the aggregation of information and eliminate the potential impact of noisy edges, we impose a restriction on the number of nodes j that can be aggregated for each central node i, using a hyperparameter p_a. By selecting only the top p_a nodes with the highest attention weights to aggregate, we effectively filter out the nodes that are least relevant to the central node, which are more likely to be connected by noisy edges. This is done to ensure that a post retains its own information and to prevent the aggregation of irrelevant or noisy information from neighboring nodes, which could degrade the performance of rumor detection. (14)${\displaystyle {\tilde{\alpha }}_i=TopK\left({\alpha }_i,p_a\right)}$

Graph structure can be difficult to distinguish for attention-based graph neural networks. Since rumor detection is formulated as a graph classification task, it is necessary to enhance the discriminative ability of our GNN. In light of the limitations of attention-based aggregators in preserving the cardinality of multisets (Zhang & Xie, 2020), we introduce modifications to the weighted summation function to address this issue. Specifically, we incorporate cardinality information into the aggregation process without altering the attention function. This approach ensures that each node in the neighborhood contributes to the central node, implicitly maintaining the cardinality information. By doing so, we not only keep the expressive power of the DynamicGAT but also improve the effectiveness of node feature representation, thereby strengthening the discriminative capabilities of the GNN. The modified equation is presented below: (15)${\displaystyle h_i^{l+1}=ReLU\left(\sum _{j\in \mathcal{N}\left(i\right)}{\tilde{\alpha }}_{ij}^l{W}^l{h}_j^l+w^l\sum _{j\in \mathcal{N}\left(i\right)}{W}^l{h}_j^l\right)}$

Here, w^l is the weight vector that is multiplied with the node features to adjust the influence of the cardinality of the multiset.

Relation-guided attention

Inspired by SuperGAT (Kim & Oh, 2021), we can leverage the presence of edges to supervise graph attention. This approach is grounded in the label-agreement assumption, which posits that the relationship between connected nodes should be closer than that between unconnected nodes. We leverage the link prediction task to provide self-supervision for attention, using labels derived from the presence or absence of edges: for a pair of nodes i and j, the label is 1 if an edge exists between them and 0 otherwise. We also consider sibling relationships and label them as y^s between 0 and 1. This mechanism, while being a part of our GAT-based model, provides additional guidance to our attention mechanism. It can be viewed as learning the graph structure because the learned node representations can reconstruct both the reply relationships between posts and the sibling relationships of posts. Hence, we refer to it as structure-aware attention. We sample a ratio of sibling edges and set this ratio as a hyperparameter η. We use DP attention with sigmoid σ to calculate the probability ϕ_ij of connection between node i and j. Note that the computation of DP attention only requires the representations of the two nodes. Therefore, we are essentially learning node representations that can reflect the graph structure. (16)${\displaystyle {\varphi }_{ij,\text{MX}}=\sigma \left(e_{ij,\text{DP}}\right)}$

Training samples consist of three parts: connected edges E, sibling edge samples E^s, and the complementary set E^c = (V × V)∖(E∪E^s). However, for graphs containing a vast number of nodes, utilizing every possible negative case in E^c would not be computationally efficient. Not all connected and sibling posts are valid, because there might be bot-generated content. Therefore, we first arbitrarily choose a total of p_n⋅|E| negative samples E⁻ from E^c where the negative sampling ratio p_n ∈ ℝ⁺ is a hyperparameter. After sampling edges, we calculate their corresponding nodes’ attention weight by dot-product attention, and select the top r ratio of edges in ascending or descending order as $\hat{E}$, E^s and ${\hat{E}}^{-}$. Due to the possibility of implicit connections between unconnected posts, we select those that are less likely to be connected to serve as negative edges. The top r ratios for each edge type are set differently to balance the importance of each type in the training process. (17)${\displaystyle \hat{E}=\text{sort}\left(E,ratio=r_c,\text{key}={\varphi }_{ij},\text{ascending}=False\right)}$ (18)${\displaystyle {\hat{E}}^s=\text{sort}\left(E^s,ratio=r_s,\text{key}={\varphi }_{ij},\text{ascending}=False\right)}$ (19)${\displaystyle {\hat{E}}^{-}=\text{sort}\left(E^{-},ratio=r_n,\text{key}={\varphi }_{ij},\text{ascending}=True\right)}$

where r_c, r_s, and r_n are the top r ratios for connected edges, sibling edges, and negative edges, respectively. These ratios are hyperparameters that can be tuned based on the specific task and dataset characteristics to achieve the best performance.

The concept behind this approach is akin to SGATs (Ye & Ji, 2023), which is capable of identifying and eliminating task-irrelevant edges in graphs, thereby producing robust outcomes even in the presence of noisy graphs. We apply a similar concept to ensure the effectiveness of SuperGAT in noisy graph environments by exclusively utilizing reliable edges. The loss function ${\mathcal{L}}_E^l$ of layer l is: (20)${\displaystyle {\mathcal{L}}_E^l=-\left(\begin{array}{c}\hfill \frac{1}{|\hat{E}\cup {\hat{E}}^{-}|}\sum _{\left(j,i\right)\in \hat{E}\cup {\hat{E}}^{-}}{1}_{\left(j,i\right)=1}\cdot log{\varphi }_{ij}^l+1_{\left(j,i\right)=0}\cdot log\left(1-{\varphi }_{ij}^l\right)\hfill \\ \hfill +\frac{1}{\left|{\hat{E}}^s\right|}\sum _{\left(j,i\right)\in {\hat{E}}^s}{\left|y^s-{\varphi }_{ij}^l\right|}^2\hfill \end{array}\right)}$

where $1_{\left(j,i\right)=1}$ and $1_{\left(j,i\right)=0}$ are indicator functions. An indicator function is a common mathematical concept used to determine whether a certain condition is true or false. These indicator functions are used to differentiate between existing and non-existing edges and calculate the loss accordingly.

Post-level attention

To address the issue of ineffective aggregation of posts with low relevance to the claim when directly applying weighted graph pooling based on the relevance between posts and the claim, we employ a gating module (Lin et al., 2021). This module is used to infuse each post with claim information, thereby leading to higher attention weights for the posts during subsequent event-level attention computations. This is a departure from previous approaches (Lin et al., 2021) that positioned the gating module before every layer of the convolution. Instead, we place this module after the last layer of the convolution. This adjustment not only bolsters the model’s ability to enhance topical coherence, but also further addresses the issue of weak relevance during subsequent graph attention pooling.

The operation of our model can be presented as follows: (21)${\displaystyle g_{c\to i}^L=sigmoid\left(W_g^L{h}_i^L+U_g^L{h}_c^L\right)}$ (22)${\displaystyle {\tilde{h}}_i^L=g_{c\to i}^L\odot h_c^L+\left(1-g_{c\to i}^L\right)\odot h_i^L}$

In this configuration, $g_{c\to i}^L$ represents the contribution of claim c to target post i in the L-th layer, consisting of trainable parameters $W_g^L$ and $U_g^L.h_c^L$ represents the hidden state of claim c in the L-th layer. ${\tilde{h}}_i^L$ represents the updated hidden state of target post i in the L-th layer after incorporating the information from the claim. For simplicity, we have omitted the bias. The ⊙ symbol denotes the Hadamard product.

Constrained event-level attention

We have made a significant adjustment to our model to address the limitations of the traditional GAT-mean-based model, particularly its potential inability to accurately weight node vectors. This mechanism works in tandem with the post-level attention, aiming to boost the semantic inference capacity of our model, with a focus on accurately capturing the relationship between the posts and the claim.

Our approach simplifies the complex joint representation used in previous models (Lin et al., 2021). Instead of employing concatenation, element-wise multiplication, and the absolute element-wise discrepancy between $h_c^L$ and $h_i^L$, which is subsequently processed using a fully-connected (FC) layer and TanH, we employ an add-attention mechanism to measure the similarity between h_c and h_i. This allows us to reduce the joint representation to a more manageable form: (23)${\displaystyle h_i^c=\left[h_c^L\parallel {\tilde{h}}_i^L\right]}$

This modification, while seemingly simple, provides performance enhancements. We empirically find that it allows the model to effectively process the relationships between posts and the claim, while providing computational efficiencies. In this setup, β_i represents the attention weight assigned to post x_i to obtain the aggregated representation $\hat{s}$ of the event. Building on this, we proceed to implement graph attention pooling on the claim-enhanced post representations. This is done to select informative posts based on inference, guided by the joint representation $h_i^c$. This leads to: (24)${\displaystyle b_i=tanh\left(FC\left(h_i^c\right)\right)}$ (25)${\displaystyle {\beta }_i=\frac{exp\left(b_i\right)}{\sum _{i}exp\left(b_i\right)}}$ (26)${\displaystyle \hat{s}=\sum _i{\beta }_i{\tilde{h}}_i^L}$

Recognizing that not all posts contribute equally to the claim, we refine our model with a thresholding step to exclude less relevant posts. Only posts with attention weights β_i exceeding a predefined threshold q are considered, ensuring that our model focuses on the most informative content when constructing the event representation $\hat{s}$. (27)${\displaystyle {\beta }_i=\frac{exp\left(b_i\right)}{\sum _{i}exp\left(b_i\right)}\left({\beta }_i>q\right)}$

This constrained event-level attention mechanism significantly refines the information filtering and processing capabilities at the graph level, which is crucial for effective rumor detection. By setting a specific threshold q, the model efficiently excludes posts that do not significantly relate to the core topic of the event, thereby ensuring that the graph representation is concentrated on the most critical information.