GFADE: generalized feature adaptation and discrimination enhancement for deepfake detection

Deepfake generation

We broadly categorize face forgery algorithms into three groups: (1) Early attempts, such as landmark-based methods (Bitouk et al., 2008; Wang et al., 2008), which use face alignment to match the source face with a similar pose to the target face, followed by adjustments to color, illumination, and other factors to blend the source and target faces. (2) Subsequent studies (Cao et al., 2013; Cheng et al., 2009; Dale et al., 2011) introduced 3D face representations, including methods like Face2Face (Thies et al., 2016), face reproduction (Suwajanakorn, Seitz & Kemelmacher-Shlizerman, 2017), and expression manipulation (Masi et al., 2016; Masi et al., 2019), which enabled more realistic face manipulation. (3) In recent years, GAN-based face-switching methods have achieved even more realistic effects. These methods not only handle complex variations in illumination, texture, and angles but also generate high-quality images, enhancing the visual realism of the forgery. For instance, the DeepFakes approach customizes a model for each target person and trains it exclusively on that person’s image data. On the other hand, FaceShifter (Li et al., 2019) can perform face swapping between any two faces without the need for prior training on the target person, thus improving the generalization and applicability of the model.

Conventional deepfake detection

In deep forgery detection research, two primary types of detectors have emerged: image-level and video-level approaches. Image-level detectors focus on identifying fake images by detecting spatial artifacts within individual frames. For instance, Face X-ray (Li et al., 2020a) uses the boundaries between altered faces and their backgrounds to identify spatial inconsistencies, while Local relation learning (LRL) (Chen et al., 2021) combines RGB image analysis with frequency-aware cues to enhance detection capabilities. However, these image-based methods have a notable limitation as they fail to exploit the temporal cues inherent in deepfake videos. In contrast, video-level detectors leverage temporal information across multiple frames to identify manipulated content. For example, LipForensics (Haliassos et al., 2021) focuses on features from the mouth region, using a model pre-trained on a LipReading dataset to capture temporal dynamics. The fully temporal convolution network (FTCN) (Zheng et al., 2021) employs 3D CNNs with a spatial kernel size of 1 to directly extract temporal information. AltFreeze (Wang et al., 2023b), on the other hand, achieves robust generalization by independently training on both spatial and temporal features, enhancing its adaptability across different datasets and deepfake techniques.

Deepfake detection generalization

With the rapid advancements in face generation and manipulation techniques, many earlier face forgery detection methods (Rossler et al., 2019; Chai et al., 2020; Dong et al., 2023a; Luo et al., 2024) struggle to handle previously unseen manipulations and datasets effectively. Moe-FFD (Kong et al., 2024b) introduces a mixture of experts model that improves face forgery detection efficiency and adaptability while reducing parameters. The task of deep forgery detection is deeply related to the generalization problem. The domain of image forgery detection has seen the development of innovative solutions from various approaches, including data augmentation (Shiohara & Yamasaki, 2022), frequency clues (Gu et al., 2022; Liu et al., 2021; Qian et al., 2020), ID information (Dong et al., 2023b), disentanglement learning (Yan et al., 2023), specifically designed networks (Zhao et al., 2021a) and 3D decomposition (Zhu et al., 2021). Video forgery detection efforts are focused on the temporal inconsistency (Wang et al., 2023b), eye blinking (Li, Chang & Lyu, 2018), and optical flow (Amerini et al., 2019).

Code repository

To promote transparency and reproducibility in research, we have made the source code of the methods described in this paper publicly available for academic research and practical reference.

The code is available on GitHub at: https://github.com/13864084796/GFADE and can also be accessed online via: https://doi.org/10.5281/zenodo.15104245.

MixStyle

Deepfake detection often faces challenges due to significant variations in visual conditions, such as differences in lighting, backgrounds, and textures across datasets. With limited datasets, models are prone to overfitting, especially when they incorrectly generalize domain-specific information from a particular dataset as class-specific features. This leads to challenges when handling cross-domain datasets, as variations in lighting, backgrounds, textures, and even differences in capturing devices and storage mediums can introduce noise. These changes can cause the model to overfit specific dataset characteristics, limiting its generalizability to unseen data. To address this issue, our study employs MixStyle (Zhou et al., 2021a), a statistical mixing technique that enhances the model’s robustness by introducing a broader range of visual conditions during training. Unlike forgery style mixing methods (Kong et al., 2024a), which operate at the forgery domain level, our MixStyle method introduces a novel approach by performing style mixing at the instance level through randomly pairing samples within the batch.This approach more precisely enhances style diversity, enabling the model not only to learn variations among different forgery styles but also to capture feature differences between real and forged samples. MixStyle is lightweight, model-agnostic, and can be seamlessly integrated into both convolutional neural network (CNN) and Transformer architectures. By disentangling style variations from core forgery features, MixStyle allows the model to focus on the intrinsic manipulation traces of deepfake content, reducing overfitting to dataset-specific styles and improving generalization to unseen forgery samples.

In the MixStyle method, we adopt a global shuffling strategy, where the entire batch is randomly reordered instead of following the original approach, which first divides the batch into multiple mini-batches and then performs shuffling and mixing within each group. Specifically, we randomly permute the batch and mix the style statistics using the newly assigned mean and standard deviation. This simplified approach reduces computational complexity while preserving the core objective of MixStyle—blending style-related feature statistics across different samples to enhance the model’s generalization ability.

MixStyle operates within the intermediate layers of the neural network, adjusting the mean and standard deviation of feature maps to mix style-related feature statistics across samples. For a given batch with feature maps f ∈ ℝ^B×C×H×W, where B represents the batch size, C is the channel count, and H and W are the height and width of the feature maps, the MixStyle process proceeds as follows:

Compute instance statistics: For each instance in the batch, the mean μ and standard deviation σ of the feature maps are computed over the spatial dimensions H and W. Specifically, for a feature map f ∈ ℝ^B×C×H×W, where B is the batch size, C is the number of channels, and H and W represent the height and width of the feature maps respectively, the instance statistics for each feature map are calculated as: (1)${\displaystyle \mu \left(f\right)=\frac{1}{H\times W}\sum _{h=1}^H\sum _{w=1}^W{f}_{:,:,h,w}}$ (2)${\displaystyle \sigma \left(f\right)=\sqrt{\frac{1}{H\times W}\sum _{h=1}^H\sum _{w=1}^W{\left(f_{:,:,h,w}-\mu \left(f\right)\right)}^2}.}$

Mix statistics between instances: After calculating the instance statistics, we randomly permute the entire batch before pairing instances for style mixing. Given two feature maps f_i and f_j, the mixed statistics $\hat{\mu }$ and $\hat{\sigma }$ are computed as follows: (3)${\displaystyle \hat{\mu }=\lambda \cdot \mu \left(f_i\right)+\left(1-\lambda \right)\cdot \mu \left(f_j\right)}$ (4)${\displaystyle \hat{\sigma }=\lambda \cdot \sigma \left(f_i\right)+\left(1-\lambda \right)\cdot \sigma \left(f_j\right)}$ where λ is a randomly sampled mixing coefficient in the range [0, 1].

Apply mixed statistics: After computing the mixed statistics, the next step is to apply these statistics to normalize the original feature maps. Specifically, for each feature map f_i in the batch, the original feature maps are normalized using the mixed mean $\hat{\mu }$ and standard deviation $\hat{\sigma }$. The normalized feature map $\hat{f}$ is computed as follows: (5)${\displaystyle \hat{f}=\hat{\sigma }\frac{f-\mu \left(f\right)}{\sigma \left(f\right)}+\hat{\mu }.}$

By incorporating mixed-style statistics from various samples, MixStyle helps the model simulate diverse visual environments during training. This approach encourages the model to learn representations that are less dependent on specific dataset styles, thereby improving its performance across different datasets and scenarios.

Additionally, high-quality forged samples are often visually similar to real samples, presenting a significant challenge in classification accuracy. MixStyle addresses this issue by reinforcing the model’s sensitivity to subtle feature differences, which enhances its ability to detect nuanced variations between forged and real samples. This method prevents the model from overfitting to specific dataset biases, significantly improving its generalization capability and overall detection accuracy in deepfake detection tasks. It is important to note that MixStyle is applied only during training to enhance the model’s generalization ability. During inference, feature statistics remain unchanged to ensure stable and consistent predictions.

ArcFace loss

One of the biggest challenges in deepfake detection is that high-quality fake samples are visually very similar to real samples, making it difficult to distinguish them with the naked eye. In such cases, where “real and fake are extremely close”, if we only use traditional binary classification loss functions (such as cross-entropy), the model is likely to focus solely on the goal of “classifying fake samples as fake”, without paying attention to how far apart real and fake samples are in the feature space. As a result, the model may learn a “fuzzy” decision boundary, and when the quality of forgeries improves further, they can easily blend with real samples and be misclassified. ArcFace loss (Deng et al., 2019) is designed to better “push apart” the distance between real and fake samples. The core idea behind its approach is as follows: We first normalize the feature vectors extracted by the model and the center vectors of each class (real/fake), and then compute the cosine similarity. For the correct class (e.g., a real sample should lie near the “real” center), an additional margin is introduced in the angle of the cosine similarity. This effectively tells the model, “In order to classify a real sample as real, not only should the cosine similarity be high, but it must also go one step further to meet the ‘acceptable’ threshold”. The final loss is computed based on this cosine similarity with the added margin, forcing the model to continuously increase the distance between different classes (real/fake) during training. Although deepfake detection is a binary classification task, the model’s discriminative capability relies not only on global classification decisions but also on constructing well-defined class boundaries in the feature space. For tasks like deepfake detection, where the boundary between real and fake samples is extremely subtle and the forgery techniques are continuously evolving, “pushing a little further” can determine whether a fake sample is accurately identified or mistakenly classified as real. ArcFace loss provides additional discriminative power at the feature level, effectively reducing the interference caused by high-quality forgeries that “lie close” to real samples, thus significantly improving the accuracy and robustness of the detection model. Especially when faced with multiple forgery techniques or cross-dataset scenarios, the larger margin established by ArcFace loss helps the model better handle unknown, high-fidelity fake samples.

In conventional classification, Softmax loss is often used, where the loss function for a given sample x_i with feature representation f_i is defined as follows: (6)${\displaystyle {\mathcal{L}}_{\text{softmax}}=-log\frac{e^{s\cdot cos\left({\theta }_{y_i}\right)}}{\sum _{j=1}^n{e}^{s\cdot cos\left({\theta }_j\right)}}}$

where:

• s is a scaling factor for numerical stability,

• θ_j denotes the angle between the feature vector and the class center.

Typically, the direct application of Softmax loss cannot effectively enforce distinct class separation in the feature space. In deepfake detection, where subtle identity cues are essential, such a loss can be insufficient for achieving the necessary level of discrimination between real and manipulated samples.

To address this, an angular margin m is introduced into the feature space, enhancing inter-class separation and reinforcing intra-class compactness. The improved loss function is given by: (7)${\displaystyle {\mathcal{L}}_{\text{ArcFace}}=-log\frac{e^{s\cdot cos\left({\theta }_{y_i}+m\right)}}{e^{s\cdot cos\left({\theta }_{y_i}+m\right)}+\sum _{j\ne y_i}{e}^{s\cdot cos\left({\theta }_j\right)}}}$

where cos(θ_yi + m) represents the calculated angular margin for the sample x_i with respect to its true class y_i. By adding an angular margin m, ArcFace loss establishes a clearer decision boundary in the feature space, making real and fake identity representations more compact and distinct.

Introducing an angular margin is particularly effective in deepfake detection, where high-quality forgeries can closely mimic real visual features, making them difficult to distinguish using traditional methods. By optimizing the angular margin with ArcFace loss, the model can more effectively identify subtle differences in identity features. Specifically, ArcFace loss enforces a separation between real and fake identities in feature space, providing clearer distinctions and reducing the risk of misclassification.

Moreover, ArcFace loss excels in generalization across diverse datasets. Deepfake generation techniques often introduce unique forgeries, each with distinct characteristics. A model without angular margin constraints may struggle to adapt to these variations, resulting in performance degradation across different forgery methods. The angular margin design, however, ensures stable class separation across a wide range of deepfake techniques, reducing the model’s reliance on specific forgery characteristics. This enhances the model’s generalization ability, improving cross-dataset and cross-technique performance. Therefore, ArcFace loss is a crucial tool for bolstering the robustness and discriminative power of deepfake detection models.

Focal loss

In deepfake detection tasks, a significant challenge arises from the imbalance between hard-to-classify and easily classified samples. This imbalance can cause the model to be biased toward the majority class, reducing its ability to accurately detect samples. Hard-to-classify samples include real faces and high-quality forged samples that closely resemble real ones in appearance. Traditional classification methods, such as those using Cross-Entropy Loss, often struggle in this scenario as they do not account for class imbalance. The standard Cross-Entropy Loss function treats all samples equally, regardless of class distribution, which can lead to performance degradation when the majority class significantly outweighs the minority class. Since more realistic forged samples are harder to distinguish, Focal Loss assigns higher weights to these samples, guiding the model to focus on subtle distinguishing features. This helps mitigate the model’s bias toward easily classified forged samples while enhancing its capability to detect complex deepfake cases.

The Cross-Entropy Loss is defined as: (8)${\displaystyle {\mathcal{L}}_{\text{CE}}=-\sum _{i=1}^N{y}_{i}log\left(p_i\right)}$

where y_i is the true label for sample i, and p_i represents the model’s predicted probability for the true class. While Cross-Entropy Loss performs effectively under balanced conditions, it may fail to focus adequately on the minority class (real samples) in imbalanced datasets.

To mitigate this issue, Focal Loss (Lin, 2017) is employed by introducing a modulating factor (1 − p_t)^γ, where p_t is the predicted probability for sample t. This factor adjusts the weight of each sample, encouraging the model to focus more on hard-to-classify samples. The Focal Loss function is defined as: (9)${\displaystyle {\mathcal{L}}_{\text{Focal}}=-\sum _{i=1}^N{\alpha }_t{\left(1-p_t\right)}^{\gamma }log\left(p_t\right)}$

where:

• α_t is a balancing factor that controls the class weight, ensuring that minority classes receive sufficient attention in the loss calculation.

• γ is the focusing parameter, which increases the weight of hard samples (those that the model finds challenging to classify) by reducing the contribution of easily classified samples.

The term (1 − p_t)^γ specifically diminishes the loss contribution from samples that the model classifies with high confidence (i.e., samples with p_t close to 1), thereby focusing more on misclassified or challenging samples. This effect directs the model’s attention toward the minority class (real samples) and helps it capture subtle differences between real and forged instances.

Focal Loss proves particularly beneficial in deepfake detection where high-quality forged samples often visually resemble real samples, making them challenging to distinguish. By enhancing focus on hard-to-classify samples, Focal Loss allows the model to prioritize subtle distinctions within real samples, thereby improving its robustness and accuracy in identifying genuine instances within a highly imbalanced dataset.

Overall framework

The proposed deepfake detection model framework starts by sampling frames from real videos, then applies self-forgery techniques (Shiohara & Yamasaki, 2022) to generate manipulated samples. During training, only real videos are utilized to construct both real and forged samples. The framework then performs multi-layer feature extraction and optimization, enhancing the model’s ability to accurately detect and robustly classify deepfake content, as illustrated in Fig. 1.

First, the model randomly samples a set of key frames from the input video to ensure that any potential manipulation traces are captured. After each frame is sampled, the model applies self-augmentation to create synthetic manipulated frames. This self-forgery simulates various operations present in a deepfake generation, allowing the model to learn a broader range of manipulation patterns and features during training. The inclusion of these augmented frames effectively increases the diversity of training samples, improving the model’s generalizability to different manipulation types.

During the feature extraction stage, the model employs a pre-trained feature extraction network (e.g., EfficientNet) to encode the self-augmented frames, generating high-dimensional feature representations. Subsequently, the model applies “the MixStyle” strategy to further improve generalization. MixStyle introduces diversity in the feature space by mixing feature statistics (e.g., mean and standard deviation) , enabling the model to adapt to variations in lighting, background, and textures. This process enhances the model’s robustness across different visual scenes, making it more stable in cross-domain and cross-dataset detection tasks.

In terms of loss function design, the model combines Cross-Entropy Loss, Focal Loss, and ArcFace loss to address class imbalance and distinguish between real and manipulated samples. Specifically:

• Cross-entropy loss: It is utilized to assess the difference between the model’s predictions and the actual labels, offering a fundamental basis for optimization in classification tasks.

• Focal loss: In situations with class imbalance, Focal Loss assigns higher weights to hard-to-classify samples, reducing the influence of easily classified samples on the total loss, thereby improving the model’s attention to minority manipulated samples.

• ArcFace loss: Introduces angular margins in the feature space to ensure a distinct separation between real and manipulated samples in the embedding space. The angular margin mechanism of ArcFace loss strengthens the model’s discriminative capability between real and manipulated identities, particularly for fine-grained face forgery detection.

The combined loss function is defined as: (10)${\displaystyle {\mathcal{L}}_{total}={\mathcal{L}}_{\text{CE}}+{\mathcal{L}}_{\text{Focal}}+{\mathcal{L}}_{\text{ArcFace}}=-\sum _{i=1}^N{\alpha }_t{\left(1-p_t\right)}^{\gamma }log\left(p_t\right)-\sum _{i=1}^N{y}_{i}log\left(p_i\right)-log\frac{e^{s\cdot cos\left({\theta }_{y_i}+m\right)}}{e^{s\cdot cos\left({\theta }_{y_i}+m\right)}+\sum _{j\ne y_i}{e}^{s\cdot cos\left({\theta }_j\right)}}}$

where ${\mathcal{L}}_{\text{CE}}$ denotes Cross-Entropy Loss, ${\mathcal{L}}_{\text{Focal}}$ denotes Focal Loss, and ${\mathcal{L}}_{\text{ArcFace}}$ represents ArcFace loss.

In each training iteration, the model generates predictions for the self-augmented frames and computes Cross-Entropy Loss, Focal Loss, and ArcFace loss. These three losses are summed to form the total loss used for backpropagation. Meanwhile, MixStyle performs style mixing in the intermediate feature space, further enhancing the model’s generalization across various visual styles.

Experimental settings

Model workflow

Cross-dataset evaluation

To demonstrate the robustness and generalizability of our approach, we perform a cross-dataset evaluation. In this process, models are trained on the FaceForensics++ (FF++) dataset, a widely acknowledged benchmark known for its extensive collection of manipulated videos, which is commonly used for deepfake detection. However, achieving high performance on the training dataset does not guarantee that the model will perform effectively on unseen data from different sources or generated by alternative deepfake techniques. To address this, we evaluate the trained models on external datasets, specifically CDF2, DFD, and DFDC, which include a diverse range of deepfake generation methods, manipulation styles, and varying video quality levels. The models’ performance is measured using three critical metrics: AUC, EER, and AP, as summarized in Table 1.

The generalized feature adaptation and discrimination enhancement (GFADE) model demonstrates excellent performance across multiple deepfake datasets (FF++, CDF2, DFD, DFDC), showcasing its strong capabilities. In terms of AUC, GFADE achieves outstanding results on all datasets, particularly reaching 94.85% on the CDF dataset, highlighting its remarkable ability to distinguish between real and fake images. In terms of EER (Equal Error Rate), GFADE also performs excellently, with values of 2.14% on FF++ and 12.92% on CDF2, indicating a low false-positive rate and high classification accuracy. For AP, GFADE reaches a high value of 99.93% on the FF++ dataset, demonstrating its powerful ability to accurately identify forged images. GFADE also exhibits excellent generalization across different datasets, further validating its wide applicability and stability.

Furthermore, the performance of the GFADE model was evaluated in terms of AUC on the DFDCP (85.39%) and FFIW (85.55%) datasets. These results demonstrate a significant improvement compared to FTCN, which achieved AUC scores of 74.0% on DFDCP and 74.47% on FFIW, underscoring the superior effectiveness of GFADE on these datasets.

Generalization performance on low-quality videos

To evaluate the robustness of our proposed GFADE model, we conducted experiments to test its generalization capabilities on low-quality video scenarios. Table 2 presents the results of our model compared to baseline methods, specifically focusing on video compression levels represented by Raw, C23, and C40 versions of the dataset.

The comparison includes three methods: FTCN, Style Latent, and our proposed GFADE. The ‘Raw’ column represents the original high-quality video, while ‘C23’ and ‘C40’ denote videos compressed at different levels, with ‘C40’ being the most compressed and hence having the lowest quality.

The experimental results demonstrate the following:

• Raw videos: Our GFADE model achieves the highest AUC score of 99.68%, outperforming both FTCN (99.5%) and Style Latent (99.6%). This indicates the effectiveness of our approach in detecting forgeries in high-quality videos.

• C23 compression level: While Style Latent slightly outperforms our model at this compression level, achieving an AUC of 99.1% compared to GFADE’s 92.57%, the results highlight the resilience of Style Latent under moderate compression. However, GFADE still demonstrates satisfactory performance, confirming its capability under such conditions.

• C40 compression level: The GFADE model significantly outperforms both FTCN and Style Latent, achieving an AUC of 76.29%. This suggests that GFADE is particularly robust under severe video compression, where details are lost, and detection becomes more challenging. The higher AUC score at this compression level reflects the effectiveness of our integration of MixStyle and advanced loss functions, which improve generalization to more challenging visual scenarios.

Overall, the results from Table 2 clearly illustrate the superior performance of our GFADE model, especially in challenging low-quality video conditions (C40), demonstrating its robustness in real-world scenarios where videos are often subject to significant compression and quality loss.

Cross-manipulation evaluation

GFADE is trained on real samples from the FaceForensics++ (FF++) dataset as well as forged samples generated using self-forgery techniques (Shiohara & Yamasaki, 2022). To evaluate the model’s generalization to other different forgery operations, we assess its performance on the four manipulation techniques provided in the FaceForensics++(FF++) dataset, including Deepfakes (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT). These diverse manipulation methods represent a wide range of deepfake generation techniques, each posing unique challenges for detection. By evaluating our model on these different manipulations, we aim to demonstrate its robustness and generalizability in detecting forged content across various manipulation types. This cross-evaluation ensures that the model is not overfitted to a specific manipulation technique and can effectively generalize to detect forgeries generated through different methods.

The results in Table 3 highlight the competitive performance of various models on cross-manipulation evaluation using the FaceForensics++ dataset. Among the listed methods, our proposed model, GFADE, achieves the best performance across most manipulation techniques, as indicated by the bold values. Specifically, GFADE attains the highest AUC scores for Deepfakes (99.99%), Face2Face (99.94%), and NeuralTextures (98.96%), demonstrating its robustness and superior generalizability. Notably, EFNB4 + SBIs achieves the best performance for FaceSwap (99.91%), marginally surpassing GFADE (99.83%). However, GFADE consistently outperforms other methods in overall performance on the FF++ dataset, achieving an AUC of 99.68%. This suggests that while some models may excel in specific manipulation techniques, GFADE provides a more balanced and robust detection capability across diverse manipulation methods.

The superior performance of GFADE can be attributed to its advanced feature extraction and generalization strategies, enabling it to effectively detect subtle artifacts across different deepfake generation techniques. Compared to existing methods such as Face X-ray and PCL + I2G, which also achieve high AUC scores, GFADE demonstrates a noticeable improvement, particularly for challenging manipulations like NeuralTextures. This indicates that GFADE is less prone to overfitting and more adaptable to various forgery patterns, which is critical for real-world applications in deepfake detection.

Further analysis

To comprehensively evaluate our model’s performance, we analyze the AUC curves and loss curves in comparison to baseline models, along with the impact of replacing different network architectures on the model’s performance.

AUC curve comparison with baseline models

In this section, we compare the AUC curves of our model with the baseline models across the training process. The AUC curves provide insight into the classification performance on the validation set, demonstrating the model’s generalization ability. Our model consistently shows higher AUC values compared to the baseline, indicating superior performance in distinguishing between fake and real samples. This improvement in the AUC curve suggests that our model achieves better classification accuracy and maintains robust generalization across validation epochs, as illustrated in Fig. 2.

The accuracy of GFADE rises rapidly in the initial phase, significantly faster than that of self-blended image (SBI). This suggests that GFADE has better convergence properties and achieves near-peak performance in fewer training iterations. In the middle and late stages of training (after about 40 epochs), the validation accuracy curve of GFADE fluctuates less and shows more stability. This implies that GFADE is more robust to perturbations in training, as shown in Fig. 2.

Loss curve comparison with baseline models

We also analyze the loss curves of our model in comparison to the baseline models throughout training (see Fig. 3). The loss curves illustrate convergence speed and stability, where our model demonstrates a faster decrease in loss during the early stages, indicating higher optimization efficiency. This rapid convergence suggests that our model is able to learn the distinguishing features between real and fake samples more effectively, reaching a stable and lower final loss compared to the baseline. Overall, the comparison of loss curves further emphasizes the efficiency and effectiveness of our approach in minimizing training loss while achieving higher detection accuracy.

Network architecture comparison

In addition to our primary experiments, we conducted a comparative analysis of various network architectures, including ResNet-50, ResNet-152, Xception, EfficientNet-b1, and EfficientNet-b4. The results, as illustrated in Fig. 4, highlight how our model performs across these architectures.The validation accuracy trends reveal clear distinctions in the convergence behavior and overall performance of the evaluated architectures. The EfficientNet series, particularly EfficientNet-b4, demonstrates a steady and rapid rise in accuracy during the early epochs, surpassing other models in achieving near-optimal accuracy. While ResNet-50 and ResNet-152 also achieve competitive results, their convergence rates are slower compared to EfficientNet-b1 and b4, indicating a potential trade-off between model complexity and optimization speed.

By evaluating our model on these different setups, we aim to determine the optimal balance between detection accuracy and computational overhead. As shown in Table 4, the experimental results highlight that our model exhibits excellent performance across all architectures, validating its strong generalization ability across different setups. This comprehensive analysis provides valuable insights into the trade-offs between computational cost and accuracy, helping us identify the most effective architecture for real-world applications. Among the evaluated architectures, EfficientNet-b4 consistently achieves the highest performance, with the best AUC scores across all test sets (FF++: 99.68%, CDF: 94.85%, DFDCP: 85.39%, FFIW: 85.55%) and an overall average AUC of 93.87%. This demonstrates its superior capability in balancing detection accuracy across diverse datasets.

Table 4:

Performance comparison of different network architectures across multiple test sets.

Bold indicates the best performance.

Architecture	Test Set AUC (%)
	FF++	CDF	DFDCP	FFIW	Avg
ResNet-50	99.16	86.90	81.83	84.85	88.19
ResNet-152	99.36	89.30	80.91	80.08	87.41
Xception	99.68	88.03	76.41	80.66	86.20
EfficientNet-b1	99.59	90.27	82.11	82.13	88.53
EfficientNet-b4	99.68	94.85	85.39	85.55	93.87

DOI: 10.7717/peerjcs.2879/table-4

Ablation study

To comprehensively assess the contribution of each component in our framework, we divided the ablation study into two main parts: the impact of different loss function combinations and the effect of MixStyle.

Visualization analysis

To intuitively understand the model’s performance and decision-making process, we employ two visualization techniques: t-SNE feature space analysis and saliency map visualization.

T-SNE feature space

This t-SNE feature space analysis allows us to observe how real and fake video samples are clustered. In this study, we present a comparative analysis of t-SNE visualizations of features extracted from FaceForensics++ datasets using GFADE and the baseline. The GFADE model demonstrates a significant improvement in feature separation, as evidenced by the distinct clustering patterns observed in the t-SNE plot shown in Fig. 5. In the GFADE t-SNE visualization, data points from different classes form clearly separated clusters with minimal overlap, indicating a strong inter-class separation. This contrasts with the baseline’s t-SNE plot, where there is substantial overlap between classes, resulting in blurred boundaries and inter-class ambiguity.

Additionally, the intra-class compactness of GFADE is evident, particularly within classes like Deepfakes (red), FaceSwap (green), and NeuralTextures (yellow), where points form tight, coherent clusters. As depicted in Fig. 5, this compactness suggests that GFADE extracts more consistent features within each class, reducing intra-class variability. Moreover, GFADE minimizes the presence of ambiguous or transition points between classes, which are prominent in the baseline’s visualization. This reduction in ambiguous points likely enhances GFADE’s reliability and robustness in real-world classification tasks, as it indicates more defined feature boundaries between real and manipulated content.

Overall, the improved feature separation and intra-class cohesion in GFADE’s t-SNE plot suggest a more effective and discriminative feature extraction process, contributing to enhanced classification performance and accuracy in distinguishing between real and manipulated videos.

Saliency map

To better interpret the model’s decision-making process, we applied saliency map techniques to visualize the regions of the input frames that significantly influence the model’s predictions. By highlighting pixel-level importance, saliency maps provide insights into the areas where the model focuses when detecting deepfake content. As illustrated in Fig. 6, our analysis shows that the model consistently attends to critical facial regions, such as the eyes, mouth, and jawline, where manipulation artifacts, such as unnatural textures or subtle misalignments, tend to appear. These visualizations not only validate the model’s attention to key areas but also emphasize the importance of local facial details in identifying deepfake forgeries.

Furthermore, GFADE exhibits greater consistency across diverse samples, consistently capturing salient facial features, as shown in Fig. 6, which enhances the model’s reliability and robustness in varied image contexts. Additionally, GFADE’s heatmaps contain less noise and show greater concentration on key regions, resulting in stronger alignment with actual facial features. This alignment offers a more intuitive and reliable visual interpretation for end-users. Collectively, these findings suggest that GFADE outperforms the baseline model in its ability to accurately localize essential features, mitigate background noise, and enhance interpretability—advantages that significantly contribute to its discriminative power and practical applicability.

Future works

While the proposed framework shows promising results, several directions remain for future improvement. First, incorporating advanced style perturbations and self-supervised learning techniques could further enhance the model’s resilience to emerging and more sophisticated forgery methods. Second, optimizing the model’s computational efficiency for real-time deepfake detection in streaming video applications remains a crucial avenue for practical deployment. Lastly, expanding the evaluation to larger, more diverse datasets and real-world scenarios will help ensure the model’s robustness and adaptability in broader contexts.

In summary, our study provides a strong foundation for improving deepfake detection, balancing accuracy, robustness, and generalization, with potential extensions into real-world, real-time applications.