GINMCL: graph isomorphism network-driven modality enhancement and cross-modal consistency learning for multi-modal fake news detection

Uni-modal methods

Uni-modal fake news detection models based on text and image information have received widespread attention. For text-based detection, traditional machine learning approaches (Zhou & Zafarani, 2019) focused on extracting semantic features at the lexical level, primarily by analyzing word frequency statistics, along with stylistic feature analysis, such as constructing sentiment lexicons to capture the emotional tone of articles (Pérez-Rosas et al., 2018) or using rhetorical structure theory to identify discourse style features (Karimi & Tang, 2019). With the development of deep learning technologies, models like RNN, LSTM, and GRU utilize hidden layer information to represent news vectors, which are then classified for detection (Ma et al., 2016). Recently, methods proposed like Vroc (Cheng, Nazarian & Bogdan, 2020) have leveraged variational autoencoders (VAE) to encode and generate text embeddings for fake news detection. Meanwhile, the Tsetlin machine (TM) (Bhattarai, Granmo & Jiao, 2022) focuses on the lexical and semantic attributes of the text. At the same time, the method proposed in Liu & Wu (2018) combines recurrent and convolutional neural networks to capture global and local variations in user characteristics along the news propagation path. Meanwhile, image information has also been explored for fake news detection.

Research has found that there are apparent distributional differences between images in real and fake news (Jin et al., 2016). Some methods detect potentially manipulated content by analyzing fraudulent patterns in images, encompassing both content and style (Khudeyer & Almoosawi, 2023). Other approaches explore detecting fake splicing information by learning image self-consistency (Huh et al., 2018). Multi-domain visual neural network (MVNN) (Qi et al., 2019) further combines image frequency and spatial domain information to identify fake images for fake news detection. Although these uni-modal approaches have progressed, they often fail to capture all the essential information from news content, neglecting the interaction between important modalities like images and social context information. This short-coming partially restricts the overall performance of the models.

Multi-modal methods

Based on the application of text and image information, extensive research has further demonstrated that integrating these two modalities significantly improves the effectiveness of fake news detection. Spotfake (Singhal et al., 2019) utilizes the pre-trained models Visual Geometry Group (VGG) and Bidirectional Encoder Representations from Transformers (BERT) to extract features separately and then concatenates their outputs for fake news detection. Spotfake+ (Singhal et al., 2020) further enhances performance by replacing BERT with XLNET. However, simple concatenation is insufficient to capture deep interactions between modalities fully. It is essential to explore more advanced multi-modal interactions to improve detection performance. To assist in detecting fake news, Event Adversarial Neural Network (EANN) (Wang et al., 2018) designed an event discriminator based on integrating text and image information. Additionally, some studies have begun focusing on the relationships between modalities. Similarity-Aware FakE news detection method (SAFE) (Zhou, Wu & Zafarani, 2020) detects the consistency between text and images and calculates their similarity to spot fake news. Another approach (Jin et al., 2017) applies an attention mechanism to enhance cross-modal understanding and classification. Multi-Modal Co-Attention Network (MCAN) (Wu et al., 2021) stacks multiple attention layers to combine text with image frequency and spatial information, imitating human news reading patterns.

Beyond the deep integration of text and images, exploring additional latent information in social news has filled many gaps in fake news detection and provided abundant inspiration for subsequent researchers. For example, Knowledge-Driven Multi-Modal Graph Convolutional Network (KMGCN) (Wang et al., 2020) constructs a heterogeneous network with text, images, and entity information, using GCN to fuse multi-modal information to generate news representations. Meanwhile, the approach in Hu et al. (2021) builds a heterogeneous graph to compare news entities with external knowledge entities for fake news detection. Research indicates that fully leveraging all available news information is essential for enhancing model performance. However, current multi-modal approaches still face challenges in fully integrating text, image, and social context information, resulting in incomplete information, which limits the overall performance of these models.

Text feature extraction

Textual information is extracted using BERT. BERT (Devlin et al., 2019) is a pre-trained language model based on a bidirectional transformer architecture. It is trained on large-scale unlabeled corpora in an unsupervised manner, achieving a deep understanding of language and demonstrating excellent performance across numerous natural language processing (NLP) downstream tasks. Specifically, given a text T, it is first tokenized into a sequence of words or subwords $\{t_1,t_2,\cdots ,t_n\}$, where each word or subword $t_i$ is mapped to a prepared token embedding vector $x_i$. BERT processes these embedding vectors through multiple layers of transformer encoders, generating an aggregated sequence representation of textual features $\{h_1,h_2,\cdots ,h_n\}$, where $h_i$ represents the hidden state vector of the $i$th word or subword.

In the modality fusion part, since different modality features involve dimensional mapping, the extracted semantic feature embedding $e_T^{\mathrm{\prime }}$ needs to be transformed through a fully connected layer to obtain a feature representation $e_T$ with a unified dimension.

Image feature extraction

For the image information I, we use the residual network (ResNet) model to extract visual features. ResNet (He et al., 2016) is a pre-trained model trained on the large-scale ImageNet dataset. It alleviates the vanishing gradient problem in deep networks by introducing residual connections. It also demonstrates outstanding performance in various computer vision tasks and possesses strong feature extraction capabilities. In detail, given an image I, it is first input into the ResNet model, which processes it through multiple layers of convolution and residual blocks to generate a set of regional feature representations $\{r_1,r_2,\cdots ,r_m\}$, where $r_i$ represents the feature vector of the $i$th region.

To prepare for multi-modal fusion, the initial image embedding $e_I^{\mathrm{\prime }}$ is transformed through a fully connected layer to obtain the final visual feature embedding $e_I$. The final visual feature embedding $e_I$ will be used for alignment and fusion with other modality information.

Social context information representation

One of the critical challenges in extracting social context information from news is effectively distinguishing different graph structure types. To effectively differentiate between social graph structures and mine deep information from social context, we utilize GIN with the WL algorithm to determine whether the extracted social graphs are topologically equivalent. The WL graph isomorphism test (Shervashidze et al., 2011; Weisfeiler & Leman, 1968) is a powerful algorithm for distinguishing various types of graph structures. The WL algorithm operates iteratively by aggregating the labels of nodes and their neighbors, and then hashing the aggregated labels into unique new labels. Figure 3 illustrates the WL test, which computes the aggregated graph for G and G′. Furthermore, if the node labels differ between the two graphs at any iteration, the algorithm determines that the graphs are non-isomorphic.

GIN leverages this highly expressive aggregation scheme to capture diverse graph structures within social context information effectively. Its injective neighborhood aggregation function can model complex graph structures, making it particularly effective in handling social context information. Specifically, GIN represents post, comment, and user nodes in the social context as a social graph structure, initializes the features of each node, and then extracts global graph features by a message-passing and aggregation process, which are used as model inputs.

Specifically, social context information is represented as a graph $G=(V,E)$, where V is the set of nodes and E is the set of edges. The three different types of nodes include post nodes $v_p$, comment nodes $v_c$, and user nodes $v_u$. The post node feature $h_{v_p}^{(0)}$ is initialized using BERT to embed the text content of the post, the comment node feature $h_{v_c}^{(0)}$ is initialized using BERT to embed the text content of the comments, while the user node feature $h_{v_u}^{(0)}$ is initialized by averaging the embeddings of the post and comments. Each node receives feature information from its neighboring nodes and updates its features by using an aggregation function to sum the features of its neighbors, combined with its own features, as shown in Eq. (2):

(2) $$h_v^{(k+1)}={\mathrm{M}\mathrm{L}\mathrm{P}}^{(k)}\left((1+{ϵ}^{(k)})\cdot h_v^{(k)}+\sum _{u\in N(v)}{h}_u^{(k)}\right)$$where $h_v^{(k+1)}$ represents the feature of node $v$ at the $(k+1)$ layer, and $N(v)$ denotes the set of neighboring nodes of $v$. For example, the neighbors of a user node $v_u$ may include the post node $v_p$ that the user has published and the comment node $v_c$ associated with the user. ${\sum }_{u\in N(v)}{h}_u^{(k)}$ represents the summation of the features of neighboring nodes. Specifically, for each node, the model aggregates the feature vectors of all its neighboring nodes. For instance, the user node $v_u$ aggregates the features of the associated post node $v_p$ and the comment node $v_c$. ${ϵ}^{(k)}$ is a learnable scalar used to adjust the weight of the node’s own features during the aggregation process. The node’s own features are weighted by $(1+{ϵ}^{(k)})$, meaning that the feature aggregation process also considers the node’s own features, adaptively assigning greater or lesser weight to them. The aggregated features are then passed through a multi-layer perceptron (MLP) for nonlinear transformation. MLP denotes a two-layer multi-layer perceptron, with each layer followed by a ReLU activation function.

Accordingly, the multi-level aggregation formula for GIN is given in Eq. (3):

(3) $$h_v^{(K)}={\mathrm{M}\mathrm{L}\mathrm{P}}^{(K-1)}\left((1+{ϵ}^{(K-1)})\cdot h_v^{(K-1)}+\sum _{u\in N(v)}{h}_u^{(K-1)}\right)$$where $h_v^{(K-1)}$ represents the node feature at the $(K-1)$th layer. To balance the trade-off between insufficient information aggregation with too few GIN layers and the difficulty distinguishing node features with too many layers, we set K to 3. Multi-level aggregation enables nodes to incorporate more information from their neighbors, capturing deeper feature representations. The final feature of a user node not only includes information about the posts it directly publishes but also captures information from other users’ comments on these posts, allowing the model to integrate a broader range of information.

After $k$ iterations, the final feature $h_v^{(k)}$ of each node already includes information from its $k$-hop neighbors. To obtain the representation of the entire graph, the above node features are aggregated into a global graph feature representation $h_G$, as shown in Eq. (4):

(4) $$h_G=\mathrm{R}\mathrm{E}\mathrm{A}\mathrm{D}\mathrm{O}\mathrm{U}\mathrm{T}\left(\{h_v^{(K)}\mid \mathrm{\forall }v\in V\}\right)$$where the READOUT function generates the entire graph’s embedding $h_G$ given the embeddings of individual nodes. READOUT denotes the graph-level representation extraction operation, which is implemented in this study as the average pooling over all node representations in the graph. A key aspect of graph-level readout is that as the number of iterations increases, the node representations corresponding to subgraph structures become more refined and global. Here, $h_v^{(K)}$ represents the final feature of node $v$ at the Kth layer, V is the set of all nodes in the social graph, and $h_G$ is the global feature representation of the social graph. During modality alignment and fusion, $h_G$ is transformed into the social graph embedding $e_G$ using a fully connected layer.

Through these steps, GIN effectively extracts deep-level information from the social graph, enhancing the performance of fake news detection.

Cross-modal consistency constraint

Features from different modalities may have significant semantic gaps. Therefore, we map the uni-modal feature embeddings extracted from text, images, and social graphs into a latent shared space, aligning features across modalities. To overcome the limitation in contrastive learning, where only positive samples are retained, and negative samples are ignored, we introduce a hard negative sampling mechanism that softens the penalty on negative samples.

First, we design a cross-modal consistency constraint to ensure close alignment of the same multi-modal features in the shared latent space. This consistency constraint is tailored for the binary classification task, aiming to predict whether modality features match based on their characteristics. Specifically, if the text embedding $e_T$, image embedding $e_I$, and social graph embedding $e_G$ are derived from the same news article S, this set is labeled as 1, indicating a positive sample; otherwise, it is labeled as 0, indicating a negative sample. Figure 4 illustrates this process in detail. The original modality embeddings are mapped to the latent shared space $\mathrm{\Omega }$ via an MLP, represented as the text embedding $e_{\mathrm{\Omega }}^T$, image embedding $e_{\mathrm{\Omega }}^I$, and social graph embedding $e_{\mathrm{\Omega }}^G$. The specific mapping to the shared space is expressed in Eq. (5):

(5) $$\begin{array}{c}{e}_{\mathrm{\Omega }}^T={\mathrm{M}\mathrm{L}\mathrm{P}}_T(e_T)\hfill \\ e_{\mathrm{\Omega }}^I={\mathrm{M}\mathrm{L}\mathrm{P}}_I(e_I)\hfill \\ e_{\mathrm{\Omega }}^G={\mathrm{M}\mathrm{L}\mathrm{P}}_G(e_G).\end{array}$$

We train the model using a consistency loss, as shown in Eq. (6):

(6) $$L_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{y}}=-\log \left(\frac{\exp \left(\mathrm{s}\mathrm{i}\mathrm{m}(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^I)\right)}{{\sum }_{j=1}^N\exp \left(\mathrm{s}\mathrm{i}\mathrm{m}(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^{I_j})\right)}\right)-\log \left(\frac{\exp \left(\mathrm{s}\mathrm{i}\mathrm{m}(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^G)\right)}{{\sum }_{j=1}^N\exp \left(\mathrm{s}\mathrm{i}\mathrm{m}(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^{G_j})\right)}\right)$$where $\mathrm{s}\mathrm{i}\mathrm{m}(\cdot ,\cdot )$ denotes the similarity measure. $\mathrm{s}\mathrm{i}\mathrm{m}(x,y)$ denotes the cosine similarity function, defined as $\mathrm{s}\mathrm{i}\mathrm{m}(x,y)=\frac{x\cdot y}{||x||||y||}$. In the formula, the numerator terms $\exp \left(\mathrm{s}\mathrm{i}\mathrm{m}(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^I)\right)$ and $\exp \left(\mathrm{s}\mathrm{i}\mathrm{m}(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^G)\right)$ represent the similarity of positive samples from the same set of multi-modal news. The denominator represents the sum of the feature embeddings. Where N denotes the total number of samples used for computing the normalized similarity. The fraction $\frac{\exp (\mathrm{s}\mathrm{i}\mathrm{m}(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^I))}{{\sum }_{j=1}^N\exp (\mathrm{s}\mathrm{i}\mathrm{m}(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^{I_j}))}$ compares the similarity of positive sample pairs to the sum of similarities of all samples. By normalizing the similarity of negative samples, the model maximizes the similarity of positive sample pairs while minimizing the similarity of negative sample pairs.

The overall loss is calculated using the negative logarithm, which amplifies the impact of positive sample pairs with lower similarity, encouraging the model to focus more on improving their alignment. By minimizing this loss function, the model gradually learns how to better align the multi-modal features from the same instance within the shared space.

Cross-modal contrastive learning

In the latent shared semantic space $\mathrm{\Omega }$, positive sample pairs are defined as multi-modal feature embeddings from the same news instance, such as $(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^I)$ and $(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^G)$, while negative sample pairs are defined as $(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^{I_{\mathrm{n}\mathrm{e}\mathrm{g}}})$ and $(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^{G_{\mathrm{n}\mathrm{e}\mathrm{g}}})$, where $I_{\mathrm{n}\mathrm{e}\mathrm{g}}$ and $G_{\mathrm{n}\mathrm{e}\mathrm{g}}$ come from different news instances. The contrastive learning loss function in the shared space can be defined as shown in Eq. (7):

(7) $$L_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}=\frac{1}{N}\sum _{i=1}^N\left[y_i\cdot \sum _{j}D(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^j)+(1-y_i)\cdot max\left(0,m-\sum _{j}D(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^j)\right)\right]$$where $j$ represents different modalities, specifically image I and social graph G. More precisely, the distance function D denotes the Euclidean distance, defined as $D(x,y)=||x-y|{|}_2$. $D(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^I)$ and $D(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^G)$ denote the distances between the positive and negative sample pairs. $y_i$ is the label of the sample pair, with a label of 1 for positive sample pairs and 0 for negative sample pairs. $m$ is the margin parameter, which ensures that the distance between negative sample pairs is greater than that between positive sample pairs. N is the total number of samples.

However, this contrastive learning method overlooks negative samples and does not fully accommodate the multi-modal fake news detection task. To enhance the precision of modality alignment, we introduce hard negative sampling. This approach softens the penalty on negative samples, providing more accurate supervision.

Hard negative sample learning

To further enhance the model’s learning effectiveness, we introduce a hard negative sampling mechanism (Robinson et al., 2021). Hard negative samples are those that are similar to positive samples in the feature space but belong to a different category. The contrastive loss function incorporating hard negative samples can be expressed as Eq. (8):

(8) $$L_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}^{\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{d}}=\frac{1}{N}\sum _{i=1}^N\left[y_i\cdot D(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^j)+(1-y_i)\cdot max\left(0,D(e_{\mathrm{\Omega }}^T,e_{\mathrm{\Omega }}^{j_{\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{d}}})-m\right)\right]$$where $e_{\mathrm{\Omega }}^{j_{\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{d}}}$ represents the negative sample embedding from either the image I or social graph G that has the highest similarity to the text embedding $e_{\mathrm{\Omega }}^T$. Here, $m$ follows the margin definition given in Eq. (7), enforcing that hard negative pairs remain at least $m$ farther from the anchor than positive pairs in the shared embedding space. In the shared space, joint training of contrastive learning and consistency loss is used to simultaneously optimize modality alignment and feature discrimination, as expressed in Eq. (9):

(9) $$L_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}=\alpha L_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{y}}+\beta L_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}^{\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{d}}$$where $\alpha$ and $\beta$ are used to adjust the importance weights of the two loss functions. They take values in the range $[0,1]$, satisfying $\alpha +\beta =1$. The coefficients are determined via a grid search over $\{0.2,0.5,0.8\}$ under the same constraint, selecting the pair yielding the highest validation F1 ( $\alpha =0.5,\beta =0.5$).

Modality fusion and classification

When incorporating the representations of the three modalities, text, image, and social graph, into the model, we leverage a cross attention mechanism (Vaswani et al., 2017) to optimize the fusion process, as each modality contributes differently to the model. First, we refine the feature representation of each modality through a self-attention mechanism. For instance, for the text modality $e_{\mathrm{\Omega }}^T$, we compute the query, key, and value matrices as detailed in Eq. (10):

(10) $$\begin{array}{c}{Q}_T=W_Q^T{e}_{\mathrm{\Omega }}^T\\ K_T=W_K^T{e}_{\mathrm{\Omega }}^T\\ V_T=W_V^T{e}_{\mathrm{\Omega }}^T\end{array}$$where $W_Q^T,W_K^T,W_V^T\in {\mathbb{R}}^{d\times d/h}$ are learnable matrices that project the input text feature $e_{\mathrm{\Omega }}^T$ into query, key, and value vectors for the self-attention mechanism. Specifically, these matrices are defined as trainable parameters during model construction and are optimized jointly with the overall multi-modal objective function. Their gradients are backpropagated from both the self-attention module and the cross-modal learning objectives, enabling the attention projections to adapt to the multi-modal fake news detection task following the standard Transformer attention learning paradigm. Here $d$ denotes the dimensionality of the input feature vectors for the text modality, and $h$ is the number of attention heads. The self-attention mechanism then computes the self-attention features of the text modality $e_T$. Initially, the attention scores are derived as shown in Eq. (11):

(11) $$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q_T,K_T,V_T)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{Q_T{K}_T^T}{\sqrt{d/h}}\right)V_T.$$

Here, the softmax function is used to normalize the attention scores, defined as $\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(x_i)=\frac{e^{x_i}}{{\sum }_{j=1}^n{e}^{x_j}}$, where $x_i$ is the $i$th element of the input vector $x=[x_1,x_2,\dots ,x_n]$. Then, the outputs from each attention head are concatenated and passed through a linear layer to produce the final self-attention feature representation, as illustrated in Eq. (12):

(12) $$e_T=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q_T^1,K_T^1,V_T^1),\dots ,\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q_T^H,K_T^H,V_T^H))W_O^T$$where $W_O^T$ is the output linear transformation matrix, and $\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(\cdot )$ denotes the concatenation of the outputs from all H attention heads. A similar process is followed for the image modality $e_{\mathrm{\Omega }}^I$ and the social graph modality $e_{\mathrm{\Omega }}^G$ to obtain their respective self-attention features $e_I$ and $e_G$. Subsequently, a cross attention mechanism is employed to capture interactions between modalities. For instance, using the text modality as the query and the image modality as the key and value, the enhanced text feature is computed as shown in Eq. (13):

(13) $$e_{TI}=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q_T,K_I,V_I).$$

Similarly, we compute $e_{IT}$, $e_{TG}$, $e_{GT}$, $e_{IG}$, and $e_{GI}$. By concatenating these features, we derive the fused multi-modal representation, as shown in Eq. (14):

(14) $$e_c=e_{TI}\oplus e_{IT}\oplus e_{TG}\oplus e_{GT}\oplus e_{IG}\oplus e_{GI}.$$

To classify the news sample S, we feed the final multi-modal representation $e_c$ into a fully connected layer, which outputs a prediction on whether S is fake news, as illustrated in Eq. (15):

(15) $$\hat{y}=\sigma (W_c{e}_c+b)$$where $\hat{y}$ denotes the predicted probability that the sample S is fake news, where $W_c$ and $b$ denote the learnable weight matrix and bias of the fully connected layer, and $\sigma (\cdot )$ represents the sigmoid activation function that maps the output to the range $[0,1]$. The model is trained using the cross-entropy loss function, as shown in Eq. (16):

(16) $$L_{\mathrm{c}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{y}}=-y\log \hat{y}-(1-y)\log (1-\hat{y}).$$

The final loss is shown in Eq. (17):

(17) $$L=L_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}+\gamma L_{\mathrm{c}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{y}}$$where $\gamma$ is a scaling coefficient that modulates the influence of the classification loss relative to the representation learning loss. It is determined via a grid search over $\{0.5,1.0,2.0\}$, selecting the value yielding the highest validation F1 ( $\gamma =1.0$).

Uni-modal models

The proposed model integrates text, image, and social context for fake news detection. To evaluate its performance, we compare it with the following uni-modal baseline models, which use each modality independently:

CNN-Text (Yu et al., 2017): CNN-Text utilizes news text information as input features. It is a convolutional neural network designed to learn news features for fake news detection. This model extracts features from news text using CNN and then inputs these features into a fully connected layer for classification.

MVNN (Qi et al., 2019): MVNN takes news image information as input features. It is a multi-view neural network that detects fake news by identifying fake images. The model captures high-level semantic information and content structure through spatial domain features and detects image tampering or misleading using frequency domain features. Attention mechanisms are used to fuse information for detecting fake images in social news.

UPFD (Dou et al., 2021): UPFD uses social context information as input features. It is a multi-layer graph neural network model that combines user preferences and external social context information. By encoding user history posts and news text, it captures user preferences and constructs a news propagation graph to extract social features. The generated news representation is then classified to detect fake news.

Multi-modal models

Multi-modal models integrate information from multiple modalities. The selected baseline models are as follows:

EANN (Wang et al., 2018): EANN is an Event Adversarial Neural Network that utilizes Text-CNN to extract textual features and VGG to extract visual features. The model concatenates textual and visual information to create a news representation, which is then fed into an event discriminator for fake news detection.

MCAN (Wu et al., 2021): MCAN is a multi-modal co-attention network that extracts semantic information and visual features from social news. It achieves cross-modal alignment and fusion by stacking multiple layers of co-attention, capturing and enhancing the interdependencies between different modal features for fake news identification.

KMGCN (Wang et al., 2020): KMGCN is a knowledge-driven multi-modal graph convolutional network that models text, visual information, and concept information from knowledge graphs within a unified semantic framework. After feature extraction using graph convolutional networks, a classifier is employed to distinguish between true and fake news.

ATT-RNN (Jin et al., 2017): Recurrent neural network with an attention mechanism (ATT-RNN) utilizes LSTM models to extract textual and social context information. An attention mechanism is used to integrate image information with text features for fake news detection.

MFAN (Zheng et al., 2022): Multi-modal Feature enhanced Attention Networks for rumor detection (MFAN) is a multi-modal feature attention network that uses attention mechanisms to integrate features from text, images, and social graphs for fake news detection.

CLFFRD (Xu et al., 2024): Curriculum Learning and Fine-grained Fusion-driven multimodal Rumor Detection framework (CLFFRD) is a multi-modal rumor detection framework that combines curriculum learning strategies with a fine-grained multi-modal fusion mechanism, and introduces a novel fine-grained fusion strategy.

Evaluation metrics

The evaluation metrics in this work include accuracy, which is used to measure the similarity between predicted fake news and actual fake news. Additionally, precision, recall, and the F1-score are utilized to assess the predictive performance for fake news detection. These evaluation metrics have been widely used in binary classification tasks.

Qualitative analysis

From the qualitative analysis, the following observations can be made:

• 1.

  w/o T: Removing text features significantly drops the model’s performance on both the Weibo and Pheme datasets, especially in accuracy and F1-score. This highlights the important role of text features in the model’s decision-making process.

• 2.

  w/o V: Excluding image features also leads to a performance decrease, though the impact is less pronounced, particularly on the Weibo dataset. This suggests that text and social graph features are more influential, with image features being slightly less important.

• 3.

  w/o G: Omitting social graph features results in a noticeable performance decline, especially on Weibo, where the F1-score falls from 0.904 to 0.830. This indicates that social graph features are vital for contextual understanding and decision-making.

• 4.

  w/o GIN: Removing the GIN module causes a performance drop, but less severe than excluding social graph features. This demonstrates GIN’s role in effectively extracting social graph features.

• 5.

  w/o CL: Excluding the contrast learning module leads to a reduction in accuracy and F1-scores, though the model remains effective. This shows that contrast learning is important for aligning and integrating features from different modalities.

• 6.

  GINMCL: The full model, which includes text, image, social graph features, GIN, and contrast learning, achieves the best performance. This confirms that the combination of these components significantly enhances the model’s effectiveness.

Quantitative analysis

From a quantitative perspective, the following conclusions can be drawn:

• 1.

  Text features: Removing text features (w/o T) leads to the most significant performance drop, with a 12.9% decrease in accuracy and 12.3% decrease in F1-score on the Weibo dataset. The Pheme dataset shows a smaller but still notable decrease of over 10%.

• 2.

  Image features: Excluding image features (w/o V) results in a relatively minor performance decline across both datasets, suggesting that text and social graph features are more critical for these tasks.

• 3.

  Social graph features: Removing social graph features (w/o G) notably impacts performance, especially on the Weibo dataset, highlighting their importance in social media information dissemination and decision-making.

• 4.

  Cross-modal consistency learning: Omitting the contrastive learning module (w/o CL) leads to a slight decrease in all metrics, indicating its role in enhancing feature representation and overall model performance.

• 5.

  GIN module: Excluding the GIN module results in some performance degradation, suggesting that GIN is more effective than traditional graph convolution methods in modeling social context information.

Discussion

The proposed GINMCL model, which integrates graph isomorphism networks with a cross-modal consistency contrastive learning mechanism, achieves superior performance on two benchmark multimodal fake news datasets. From the experimental results and observations of model behavior, we derive the following insights:

First, the cross-modal consistency modeling mechanism significantly enhances the synergistic understanding among modalities, demonstrating improved robustness especially in cases of semantic mismatch between text and images. Second, the incorporation of social structure strengthens the modeling of information hierarchies by capturing user-level propagation, although its benefits are limited when the propagation graph is sparse or the event granularity is small. Analysis of misclassified samples further reveals limitations of the model in handling modality conflicts and information incompleteness. Additionally, the model’s performance noticeably degrades under incomplete modality inputs, indicating that the current architecture still relies on multimodal completeness. Future work could explore modality availability assessment mechanisms to improve generalization.

Limitations

Although the model performs well overall, it still has several limitations. First, GINMCL is sensitive to the completeness of multimodal inputs; if the image or social graph is missing or of low quality, the model’s performance may degrade significantly. Second, the incorporation of graph neural networks and contrastive learning mechanisms increases the model’s complexity and computational cost to some extent, limiting its suitability for real-time scenarios. Third, the current validation is conducted only on two medium-scale binary classification datasets; generalization tests on multi-label, large-scale, or cross-lingual datasets have not yet been performed, and the model’s universality requires further evaluation. Future work will focus on addressing these issues.