AdvFaceGAN: a face dual-identity impersonation attack method based on generative adversarial networks

Generator adversarial network

Goodfellow et al. (2014) proposed generative adversarial networks, constructing fully-connected generator and discriminator networks. Through alternating training of Generator G and Discriminator D, the framework ultimately enables the Generator G, taking noise z as input, to produce samples matching the data distribution. However, fundamental issues in original GANs remained unresolved: training instability, inadequate guidance from loss metrics for training progression, and lack of diversity in generated samples.

Subsequent research efforts addressed these challenges. DCGAN (Radford et al. 2016) implementing convolutional architectures for both generator and discriminator. The convolutional layers effectively capture spatial structures and local features in images, significantly improving performance in image generation tasks. Wasserstein generative adversarial networks (WGAN) (Arjovsky & Bottou, 2017) theoretically demonstrating that the instability of traditional GANs stems from the Jensen–Shannon (JS) divergence becoming ineffective when the supports of distributions are disjoint or minimally overlapping, leading to gradient explosion or vanishing. Their solution employed Wasserstein distance as an alternative metric, which measures the minimum “transport cost” between distributions and provides smooth gradients even with disjoint supports. The practical implementation involved enforcing 1-Lipschitz constraints on the discriminator through weight clipping, leveraging Kantorovich-Rubinstein duality to transform Wasserstein distance optimization into a dual problem. WGAN’s modifications included: removing the final Sigmoid layer in the discriminator, eliminating logarithmic computations in loss functions, applying hard weight clipping with constant C, and recommending RMSProp over momentum-based optimizers. These changes substantially improved training stability, mitigated mode collapse, and ensured sample diversity. Nevertheless, weight clipping introduced two limitations: restricted network capacity due to parameter constraints, and potential gradient explosion/vanishing issues.

This led to the development of WGAN-GP (Gulrajani, Ishaan et al. 2017), which introduced a gradient penalty term in the discriminator loss. This penalty enforces soft constraints on the gradient norm of discriminator outputs for both real and generated samples, maintaining smooth gradients without explicit weight clipping. The gradient penalty approach preserved 1-Lipschitz continuity while avoiding clipping-induced performance degradation, enabling higher-quality image generation and compatibility with momentum-based optimizers like Adam for faster convergence.

Adversarial attacks

This section reviews the development of adversarial attack techniques and related research on adversarial attacks for face recognition tasks. The methods for generating adversarial samples were first proposed in the field of image classification models, primarily including optimization-based methods, gradient-based methods, and generative adversarial network-based methods:

• Optimization-based methods: For example, C&W (Carlini & Wagner, 2017) improved the target optimization function in iterative attacks and gradually constrained the function within the disturbance magnitude limits, effectively reducing the perturbation size of adversarial samples while maintaining excellent visual quality.

• Gradient-based methods: For example, FGSM (Goodfellow, Shlens & Szegedy, 2014) used first-order gradients to efficiently compute adversarial perturbations. Although fast, this method tends to be unstable in its attack. Later methods such as PGD (Madry et al., 2017), BIM (Kurakin, Goodfellow & Bengio, 2018), and MI-FGSM (Dong et al., 2018) introduced iterative optimization steps, which enhanced the attack effectiveness of adversarial samples.

• GAN-based methods: For instance, Baluja & Fischer (2018) proposed the ATN network, summarizing two patterns for generating adversarial samples: P-ATN and AAE. The P-ATN pattern advocates training a residual module to generate additive perturbations with p-norm constraints over the source face. This preserves the overall information of the source face, but the perturbations are more apparent on the edges or corners of the image. The method in this paper is a variation of the P-ATN pattern. AdvGAN (Xiao et al., 2018), a representative work of the P-ATN pattern, determined the basic architecture for the Generator, Discriminator, and surrogate models, and introduced a hyperparameter ɛ into the perturbation loss, which can control the perturbation magnitude by adjusting ɛ.

The generation of adversarial examples for face recognition models includes traditional attack methods such as Fast Gradient Sign Method (FSGM), Momentum Iteration—Fast Gradient Sign Method (MI-FGSM), and Carlini & Wagner (C&W), which adapt surrogate models from image classification models to face recognition models to generate adversarial faces. Or newly developed attack methods for face recognition models, which can be further categorized into two approaches: local perturbation methods, which are more suited for physical-domai, and global perturbation methods, which are more applicable in the digital-domain.

• Local perturbation-based methods: AdvGlass (Sharif et al., 2016) applied an iterative gradient descent algorithm to generate adversarial glasses and incorporated smoothing and printer color gamut losses to make the adversarial glasses printable for physical-world attacks; AdvHat (Komkov & Petiushko, 2021) used the MI-FGSM method to generate adversarial hats; Adv-MakeUP (Yin et al., 2021) used GANs to generate adversarial makeup with good transferability but limited attack effectiveness; and in 2023, Yang et al. (2023) at the Conference on Computer Vision and Pattern Recognition (CVPR) conference proposed the design of adversarial textured 3D masks shortened as Adversarial Textured 3D meshes (AT3D) with complex topological structures, which were 3D printed and placed on an attacker’s face to evade defenses. The perturbations were generated using BIM and updated in the low-dimensional coefficient space of 3DMM. AT3D has the best attack effectiveness among local perturbation methods, but its concealment is still weak, as the mask has clearly visible edges. Both AT3D and Adv-MakeUP will be included as representative local perturbation methods for comparison in subsequent sections.

• Global perturbation-based methods: AdvFaces (Deb, Zhang & Jain, 2020) employed FaceNet as a surrogate model under the P-ATN pattern to generate high-quality adversarial faces for obfuscation or impersonation attacks. TIP-IM (Yang et al., 2021) enhanced the naturalness of adversarial faces by introducing a maximum mean discrepancy (MMD) loss based on MI-FGSM, and optimized attack efficacy through a greedy insertion algorithm targeting multiple facial identities. AMT-GAN (Hu et al., 2022) addressed the conflict between adversarial noise and cycle-consistency loss via a regularization module and joint training strategy, achieving notable performance improvements in black-box scenarios. Sibling-Attack (Li et al., 2023) leverages gradient information from facial attribute recognition tasks and integrates PGD-based perturbation generation to substantially boost attack success rates and transferability. DiffAM (Sun et al., 2024), as an enhancement to the Adversarial Makeup Transfer-GAN (AMT-GAN) framework, synergized the high-quality generation capability of diffusion models with CLIP’s fine-grained semantic control, demonstrating superior black-box attack success rates and naturalness compared to existing makeup transfer methods. AdvFaces, TIP-IM, Sibling-Attack, and DiffAM will be included as representative global perturbation methods in our comparative experiments.

Threat model

Attacker: The insider employee uses open-source pre-trained face recognition models, open-source datasets, etc., to build adversarial face generation technology. The insider employee is unaware of the model structure and parameters of the face recognition system or the datasets used for training. However, the insider employee has an image of the attacker’s face.

Attack scenario: Some face recognition systems, such as those used in attendance machines, allow administrators to upload face images of employees for registration. Administrators can request employees to submit their face images online and upload them to register or update personnel information.

Attack goal: The attacker is identified as the insider employee and is able to pass through the face recognition system without disrupting the insider employee’s normal authentication.

Limitations: The adversarial face must be uploaded to the face database.

Attack process: As shown in Fig. 1A, when the attacker’s face, Attacker_2, is compared to the three clean faces of the insider employee in the face database (Victim_2, Victim_3, and Victim_4), the feature similarity is all below 15% (as returned by the Aliyun Face Search API), which is far below the default threshold of 67.87% for Aliyun Face Search API, so direct matching with the insider employee fails. However, in Fig. 1B, the insider employee uses the AdvFaceGAN method in this paper to generate a “dual-identity” adversarial face, FakeFace, based on the attacker’s face, Attacker_1, and one of the insider employee’s faces, Victim_1. This adversarial face is submitted to the Administrator. Due to the high visual quality of FakeFace, it can pass the Administrator’s human review, and also pass defense against the upload when compared to the source face quality. Additionally, the feature similarity between FakeFace and the three existing faces of the insider employee in the database is 72%, 73%, and 69% (as returned by the Aliyun Face Compare API), which surpasses the 69% within-class similarity threshold recommended by the Aliyun AddFace API. As a result, FakeFace is successfully uploaded to the face database under the insider employee. When the attacker attempts to attack face recognition again, Attacker’s face Attacker_2 will have a 70% feature similarity to FakeFace, which is associated with the insider employee. Therefore, the attacker will be recognized as the insider employee and pass through the face recognition system, Meanwhile, during this period, the insider employees are able to pass the face recognition system normally.

It can be observed that under this threat model, the various liveness detection defenses set up during the face image capture in the face recognition system are cleverly bypassed. In contrast, some similar advanced attack methods either fail due to being too obvious to pass the Administrator’s visual inspection (e.g., SiblingAttack, AT3D), or they cannot be successfully uploaded to the face database under the insider employee’s profile due to insufficient similarity to the source identity (e.g., SiblingAttack, AdvFaces), or they fail to match the adversarial face with the target identity due to insufficient similarity (e.g., AT3D), even if the upload is successful. Furthermore, if we assume the database Administrator is an insider or there is some vulnerability that grants the Attacker direct permission to upload faces to the face database, any employee in the database—even the CEO—could unknowingly and imperceptibly be impersonated by the attacker as an “insider.”

AdvFaceGAN structure

The overall structure of AdvFaceGAN, as shown in Fig. 2, consists of three components: a Discriminator D, a Generator G, and auxiliary discriminators FRs. The interplay between these components allows the adversarial perturbation ϵ generated by G to be added to the attacker’s face x, resulting in the adversarial face $\tilde{x}$, which remains in the same distribution as the real face x. Additionally, $\tilde{x}$ has enough similarity to both the attacker’s face x and the victim’s face y.

The detailed architecture of Discriminator D is illustrated in Fig. 3. In the construction of the discriminator, AdvFaceGAN employs multiple residual convolutional layers for feature extraction and utilizes a final full connected layer to map the extracted features into a scalar output representing image authenticity. However, the introduction of gradient penalty imposes a critical architectural constraint: conventional batch normalization (BN) layers—commonly used in standard GANs and WGANs—are incompatible with this framework.

This incompatibility arises because gradient penalty requires a strictly sample-wise input–output relationship for accurate gradient calculation, whereas BN normalizes statistics across batch dimensions (i.e., batch inputs → batch outputs). Such batch-level operations disrupt the per-sample gradient computation essential for effective gradient penalty enforcement. To resolve this conflict while maintaining normalization benefits, our method replaces BN with instance normalization (IN) in the discriminator.

The detailed architecture of Generator G is illustrated in Fig. 4. In the generator architecture, AdvFaceGAN constructs an autoencoder integrated with residual blocks to synthesize adversarial perturbations ϵ. When processing an input pair comprising a source facial image x and a target facial image y, the Generator first concatenates x and y channel-wise. This concatenated input is then fed into an encoder module that progressively downscales the feature maps through multiple strided convolutional layers. The compressed latent representation subsequently passes through a series of residual blocks to learn deep-level discriminative features. Following this, a decoder module employing transposed convolutional layers gradually upscales the features to the original spatial dimensions. Finally, the output undergoes normalization via Tanh activation layer, producing an adversarial perturbation ϵ with identical dimensions to the input facial images.

Ensemble of face recognition models

In previous research, researchers typically computed the loss by using a single target model or by averaging multiple models. In our approach, inspired by the findings of M3D (Zhao et al., 2023), we reconsider the role of surrogate models in the generation of adversarial face samples. We introduce multiple face recognition surrogate models with maximal model differences to construct an ensemble white-box model, FRs. FRs acts as an auxiliary discriminator in the adversarial training process between the Generator and the Discriminator, enhancing the Generator’s ability to produce adversarial samples with a better generalization error bound. The principle is illustrated in Fig. 5.

As seen in Fig. 5A, when using a single surrogate model, the abstract decision boundary of the trained adversarial face (red dashed line) and the generalization error to the black-box model (red shaded area in a3) are significantly larger due to the surrogate and black-box model differences. This results in a notable loss in attack success rates. However, in Fig. 5B, when using multiple ensemble white-box models, the abstract decision boundary of the trained adversarial face becomes more aligned with the black-box model, reducing the generalization error (red shaded area in b3), thereby improving the effectiveness of black-box attacks.

Training process design

In order to ensure stable convergence during the training process, we follow the recommendations of WGAN-GP and design the model structure and training details as follows: (1) adopting the Adam optimizer to improve performance; (2) enforcing a gradient penalty term with coefficient λ =10 in the discriminator’s loss function (Eq. (1)), ensuring 1-Lipschitz continuity between real and generated samples; (3) eliminating log-transformations in the loss calculations; (4) replacing BatchNorm layers with InstanceNorm in both the Generator G and Discriminator D architectures to ensure the effectiveness of the gradient penalty.

In order for the Discriminator D to learn to distinguish between adversarial faces and real faces, the loss function L_D is calculated as shown in Eq. (1): (1)${\displaystyle L_D=\underset{x\sim P_g}{E}\left[D\left(x\right)\right]-\underset{x\sim P_r}{E}\left[D\left(x\right)\right]+\lambda \underset{\hat{x}\sim \chi }{E}\left[{\left({∥\nabla \hat{x}D\left(\hat{x}\right)∥}_2-1\right)}^2\right]}$ (2)${\displaystyle \hat{x}=\alpha x_r+\left(1-\alpha \right)x_g\left(\alpha \sim U\left[0,1\right],x_r\sim P_r,x_g\sim P_g\right)}$

where P_g represents the space of adversarial samples, P_r represents the space of real samples, and $D\left(x\right)$ is the prediction value of the discriminator D for the face x. The final part is the gradient penalty term, where λ is a hyperparameter controlling the strength of the penalty, set to 10. χ is the overall sample space containing both the adversarial sample space P_g and the real sample space P_r. The collection method for the intermediate sample $\hat{x}$ in the overall sample space is given by Eq. (2). Here, $\nabla \hat{x}D\left(\hat{x}\right)$ is the gradient of the discriminator for the intermediate sample $\hat{x}$.

For adversarial faces, we want $D\left(x\right)$ to be as small as possible, hence the positive sign in front of $E_{x\sim P_g}\left[D\left(x\right)\right]$; for real faces, we want $D\left(x\right)$ to be as large as possible, hence the negative sign in front of $E_{x\sim P_r}\left[D\left(x\right)\right]$. The numerical space of the discriminator’s gradient is the overall sample space, which is of high dimensionality and difficult to compute. Therefore, we sample real and adversarial intermediate samples $\hat{x}$ from each batch to compute the penalty term: $\underset{\hat{x}\sim \chi }{E}\left[{\left({∥\nabla \hat{x}D\left(\hat{x}\right)∥}_2-1\right)}^2\right]$. This ensures that the L₂ norm of $D\left(\hat{x}\right)$ is as close to 1 as possible, maintaining the Lipschitz continuity of the discriminator function. Optimizing L_D allows the discriminator to learn to differentiate between adversarial faces and real faces.

In order for the Generator G to learn to generate adversarial face samples with high attack capability and high stealthiness, the Generator’s loss function is obtained by adding four types of losses. The convergence goal of the model is controlled by setting weight hyperparameters for the losses. L_G’s calculation is shown in Eq. (3): (3)${\displaystyle L_G=L_{GAN}+{\lambda }_{pert}{L}_{Pert}+{\lambda }_{adv}{L}_{adv}+{\lambda }_{st}{L}_{st}.}$

In the design of the Generator’s loss function, we incorporate four types of losses to form the total loss. These include the adversarial loss L_gan from the discriminator D. By fixing the parameters of discriminator D and optimizing the Generator according to L_gan, we ensure that the adversarial face $\tilde{x}=x+G\left(x\right)$ is closer to real face’s distribution. The calculation of L_gan is given by Eq. (4): (4)${\displaystyle L_{gan}=-\underset{x\sim P_r}{E}\left(D\left(x+G\left(x\right)\right)\right).}$

The perturbation loss L_Pert from the adversarial perturbation ϵ = G(x, y), doing the perturbation amount calculation with two-paradigm numbers can effectively penalize the perturbation at each pixel point so as to avoid useless perturbation, and control the perturbation amount by max to be unpunished at low perturbation upper limit ɛ, thus encouraging the learning of perturbation and accelerating the model convergence. The computation of L_pert is shown in Eq. (5): (5)${\displaystyle L_{pert}=\underset{x,y\sim P_r}{E}\left[\underset{}{max}\left(\varepsilon ,{∥ϵ∥}_2\right)\right]=\underset{x,y\sim P_r}{E}\left[\underset{}{max}\left(\varepsilon ,{∥G\left(x,y\right)∥}_2\right)\right]}$

where ${∥G\left(x\right)∥}_2$ indicates the perturbation amount calculated from ϵ = G(x) using the ${∥\cdot ∥}_2$ norm, and ɛ denotes the upper limit on perturbation.

The structural loss L_st from the structural similarity between the source face x and the adversarial face $\tilde{x}=x+G\left(x,y\right)$ and is controlled by max so that the structural similarity is not penalized when it is higher than the lower limit of the structural similarity ζ. L_st is computed as in Eq. (6): ${\displaystyle L_{st}=\underset{x\sim P_r,\tilde{x}\sim P_g}{E}\left[\underset{}{max}\left(1-\zeta ,1-ssim\left(x,\tilde{x}\right)\right)\right]}$ (6)${\displaystyle =\underset{x,y\sim P_r}{E}\left[\underset{}{max}\left(1-\zeta ,1-ssim\left(x,x+G\left(x,y\right)\right)\right)\right]}$

where ssim(⋅, ⋅) denotes the structural_similarity_index_measure function provided by torchmetrics and ζ denotes the lower limit of structural similarity.

The identity loss L_adv from the auxiliary discriminator FRS combines the source identity loss L_{adv_source} and the target identity loss L_{adv_target}, statistically predicts the losses of multiple face recognition models with the largest differences, and by adjusting the source identity loss weights η, it can make the generated antagonistic faces have a balanced “dual identity”, L_adv is calculated as in Eq. (7): ${\displaystyle L_{adv}=\eta \ast L_{ad{v}_{source}}+L_{ad{v}_{target}}}$

    $=\underset{x,y\sim P_r,\tilde{x}\sim P_g}{E}\left[\eta \ast \frac{{\sum }_{i=1}^M\left(1-cos\left[f_i\left(\tilde{x}\right),f_i\left(x\right)\right]\right)}{M}+\frac{{\sum }_{i=1}^M\left(1-cos\left[f_i\left(\tilde{x}\right),f_i\left(y\right)\right]\right)}{M}\right]$ (7)${\displaystyle =\underset{x,y\sim P_r}{E}\left[\eta \ast \frac{\sum _{i=1}^M\left(1-cos\left[f_i\left(x+G\left(x,y\right)\right),f_i\left(x\right)\right]\right)}{M}\right.+\left.\frac{\sum _{i=1}^M\left(1-cos\left[f_i\left(x+G\left(x,y\right)\right),f_i\left(y\right)\right]\right)}{M}\right]}$

where x denotes the source face, $\tilde{x}=x+G\left(x,y\right)$ denotes the adversarial face, y denotes the target face, M denotes the number of white-box substitution models integrated by FRS, f_i(⋅) denotes the feature vector computation function corresponding to the ith fixed-parameter white-box substitution model, and cos(⋅, ⋅) denotes the cosine similarity computation function of two features.

The pseudocode for the training process of AdvFaceGAN is shown in Table 1.

Evaluation metrics

In order to reflect the advantages of the adversarial faces generated by the method in this paper in terms of visual quality and attack effect, we design the following evaluation metrics:

Visual quality metrics: we take the structural similarity (SSIM) peak signal-to-noise ratio (PSNR), mean squared error (MSE) of the source face and the adversarial face as the visual quality evaluation metric. The range of structural similarity is [0, 1], when two images are the same, the value is 1. The larger the image structure gap, the lower the value, and it is generally considered to be greater than 0.92 to satisfy the visual similarity. The peak signal-to-noise ratio evaluates the image quality, the larger the better, greater than 30dB, the human eye is difficult to detect the difference between the compressed and the source face. MSE intuitively indicates the magnitude of the image change, the smaller the weaker the change. The specific formulas for the three indicators are shown in Eqs. (8), (9) and (10): (8)${\displaystyle \text{SSIM}\left(x,y\right)=\frac{\left(2{\mu }_x{\mu }_y+c_1\right)\left(2{\sigma }_{xy}+c_2\right)}{\left({\mu }_x^2+{\mu }_y^2+c_1\right)\left({\sigma }_x^2+{\sigma }_y^2+c_2\right)}}$ (9)${\displaystyle PSNR=20{log}_{10}\left(\frac{MA{X}_I^2}{\sqrt{MSE}}\right)}$ (10)${\displaystyle MSE=\frac{1}{mn}\sum _{i=0}^{m-1}\cdot \sum _{j=0}^{n-1}{\left[I\left(i,j\right)-K\left(i,j\right)\right]}^2}$

where, x, y are comparison images, μ_x, μ_y, ${\sigma }_x^2$, ${\sigma }_y^2$ are the mean and variance of x and y, ${\sigma }_y^x$ is the covariance of x and y, $c_i={\left(k_{i}L\right)}^2\left(i=1,2\right)$ is the constant used to maintain the stability, where k₁ = 0.01, k₂ = 0.03, and L is the range of pixel values of the image. $I\left(i,j\right)$ is the adversarial face, K(i, j) is the source face, MAX_I is the maximum possible pixel value of the image and mn is the size of the image.

Attack effectiveness metrics: We use the attack success rate (ASR) of the attack on the face recognition model as the evaluation metrics of the attack effectiveness, and the ASR aims at calculating the ratio of the successful realization of the attack among N pairs x_s ∼ x_t. In the traditional impersonation attack, the cosine similarity between the antagonist face features and the target non-homo face features is greater than the recognition threshold is considered as successful impersonation, which is calculated as shown in Eq. (11): (11)${\displaystyle AS{R}_1=\frac{\sum _i^N{1}_{\tau }\left(cos\left[f\left(x_i+G\left(x_i,y_i\right)\right),f\left(y_i\right)\right]>\tau \right)}{N}\times 100\text{\%}}$

where 1_τ is the indicator function, x is the source face, y is the target face, $G\left(\mathrm{\ast }\right)$ is the pre-trained generator, f( ∗) is the target black-box face recognition model, f(x) is the output face features after inputting the face x, cos[a, b] denotes the computation of the cosine similarity between the two face features, and τ is the recognition threshold of f at 0.1% FAR or 0.01% FAR (False Accept Rate) under the recognition threshold. N is the total number of image pairs.

This article found that the face recognition system database often requires that the uploaded face and other faces in the person’s face database are higher than a certain threshold, so it further explores the success rate of the attack in the case where the antagonist face is required to maintain a high degree of similarity with the source identity, i.e., the cosine similarity between the antagonist face features and the source same-person face features and the target nonhomo-person face features are both greater than the recognition threshold is considered to be successful in the impersonation, and its formula is shown in Eq. (12) shows: (12)${\displaystyle AS{R}_2=\frac{\sum _i^N{1}_{\tau }\left(cos\left[f\left(x_i+G\left(x_i,y_i\right)\right),f\left(x_i\right)\right]>\tau \&\&cos\left[f\left(x_i+G\left(x_i,y_i\right)\right),f\left(y_i\right)\right]>\tau \right)}{N}\times 100\text{\%}.}$

However, only using the above two attack success rates as the attack effect evaluation metric is still insufficient, because whether the attack is successful is based on whether the feature similarity is greater than the model’s recognition threshold, which is a discrete metric. Assuming that the recognition threshold of the face recognition model is 30%, an antagonistic sample with 70% similarity to the target face and an antagonistic sample with 50% similarity to the target face are equivalent in the statistics of ASR₁, however, it is obvious that the former achieves a better attack effect. Therefore, when evaluating the attack effect, we add the average similarity of the output of the target face recognition model, which is a continuous metric that can compare the attack effect more effectively. The average similarity with the source identity (Fake & Source Similarity, FSS) and the average similarity with the target identity (Fake & Target Similarity, FTS) under the face recognition model f are shown in Eqs. (13) and (14). (13)${\displaystyle FSS=\frac{\sum _i^{N}cos\left[f\left(x_i+G\left(x_i,y_i\right)\right),f\left(x_i\right)\right]}{N}\times 100\text{\%}}$ (14)${\displaystyle FTS=\frac{\sum _i^{N}cos\left[f\left(x_i+G\left(x_i,y_i\right)\right),f\left(y_i\right)\right]}{N}\times 100\text{\%}.}$

Experimental setup

Dataset: Our training set employs CASIA-WebFace (Yi et al., 2014), comprising 455,594 facial images of 10,575 distinct individuals after removing low-quality samples (blurred, occluded, or misaligned faces). We allocated 10% of the remaining data (stratified by identity) as the validation set. Our test set employs LFW (Huang et al., 2008), containing 13,233 images from 5,749 unique identities.

All samples across training, validation, and test sets underwent standardized preprocessing: (1) Face detection and alignment using MTCNN with five keypoints (eyes, nose, mouth corners), followed by (2) resizing to 112 ×112 pixels through bicubic interpolation to maintain consistency with input specifications of mainstream open-source face recognition models (e.g., FaceNet, ArcFace).

Model information: In the training and testing process of AdvFaceGAN, multiple pre-trained FRModels by the GitHub project face-robustness-benchmark, and three commercial face recognition API models are used. The basic information of these models, the recognition threshold at 0.1%FAR, and the actual TAR and FAR calculated by using the training set CASIA-WebFace in this article to compare 6,000 random faces of homogeneity and non-homogeneity at each model recognition threshold can be described in Table 2.

Baseline methods: This experiment compares three traditional gradient or optimization-based baseline methods FGSM (Goodfellow, Shlens & Szegedy, 2014), MI- FGSM (Kurakin, Goodfellow & Bengio, 2018) and C&W (Carlini & Wagner, 2017); locally perturbed baseline methods: Adv-MakeUP (Yin et al., 2021) and AT3D (Yang et al., 2023); globally perturbed baseline methods: AdvFace (Deb, Zhang & Jain, 2020), TIP-IM (Yang et al., 2023), SiblingAttack (Li et al., 2023) and DiffAM (Sun et al., 2024).

The principles followed when reproducing these comparison methods are: use the parameters from the original paper’s comparison experiments, or adjust the comparison methods to generate adversarial samples with a L₂ norm perturbation that is close to the perturbation of the adversarial samples generated by the AdvFaceGAN model, which is 5. The specific model settings are as follows:

• FGSM, MI-FGSM: Use the implementation from the GitHub project ShawnXYang/Face-Robustness-Benchmark. The distance metric is set to the L₂ norm, and the perturbation control parameter is set ɛ = 6, with other parameters as default. After verification, the adversarial samples generated with ɛ = 6 have a L₂ norm perturbation close to 4.6, similar to the samples generated by our AdvFaceGAN model.

• C&W: Use the implementation from the GitHub project ShawnXYang/Face-Robustness-Benchmark. The perturbation control parameter is set ɛ = 16, with other parameters as default, and the code is modified to adjust the optimization objective to achieve 1.5 times the recognition threshold on the white-box surrogate model’s FTS, thereby enhancing the black-box attack capability.

• AdvMakeUP: Use the implementation provided in the paper author’s GitHub project Tencent/TFace. All parameters remain unchanged, i.e., using the parameters from the original article for comparison experiments. However, the ensemble model used in the original article (irse50, facenet, and mobile_face) is modified to the models used by AdvFaceGAN (ArcFace, FaceNet, and ResNet50), and the dataset is changed from the original makeup dataset to the LFW dataset (using Face++’s face detection API to create the dataset keypoint files required by the code).

• AT3D: Use the implementation from the author’s GitHub open-source project thu-ml/AT3D. Refer to the AT3D-P parameter settings from the original paper’s digital-domain comparison experiment. The mask type is set to glasses + nose, with 300 iterations, and the perturbation control parameter is set η = 5 slightly higher than 3 (for stronger attack power). The perturbation generation algorithm is BIM.

• AdvFaces: Use the implementation from the paper author’s GitHub project ronny3050/AdvFaces. Using the parameters from the original paper for comparison experiments, with the perturbation control parameter ɛ = 8 as stated. However, the original FaceNet model is replaced with the FaceNet model used by AdvFaceGAN.

• TIP-IM: Use the implementation from the paper author’s GitHub project ShawnXYang/TIP-IM. All parameters remain unchanged, and AdvFaceGAN’s ArcFace, FaceNet, and ResNet50 models are used as surrogate models to generate adversarial samples.

• SiblingAttack: Use the implementation from the paper author’s GitHub open-source project Tencent/TFace. All parameters remain unchanged, i.e., using the parameters from the original article for comparison experiments. After verification, the adversarial samples generated with these parameters have a L₂ norm perturbation close to 10.9.

• DiffAM: Use the implementation from the author’s GitHub project HansSunY/DiffAM. The ensemble model used is changed from the original article’s irse50, facenet, and mobile_face to AdvFaceGAN’s ArcFace, FaceNet, and ResNet50, and the adversarial samples are generated using parameters that match the attack power mentioned by the author in the issues section.

Experimental details: In this experiment, the size of both the input source face and the output adversarial face is 3 × 112 ×112, and the batch size is set to 8; the Adam optimizer is used for training with parameters β₁ of 0.5, β₂ of 0.999, and the learning rate is set to 1e−4; the hyperparameter λ_pert = 1, λ_adv = 10 and λ_st = 20, which will make the various loss values equalize in magnitude, and is conducive to the effective convergence of the trained model on each loss.

Loss ablation and parametric experiments

This section analyzes the significance of the perturbation loss L_pert, structural loss L_st, and source identity loss L_{adv_source} involved in the methodology of this paper, and constructs ablation experiments to analyze the specific effects they play. We also propose the specific numerical determination methods for the hyperparameters related to each loss: the upper limit of perturbation ɛ, the lower limit of structural similarity ζ, and the weight of source identity loss η.

Ablation of L_pert and determination of ɛ

First, we tried λ_pert = λ_st = μ = 0. After training for 500 epochs, the model converged, but the Generator G chose to directly replace the source face with the target face. At this point, both the SSIM and MSE metrics were poor, and the FSS was very low, deviating from the premise that adversarial samples should maintain high similarity with the source face.

We sought to determine the value at which the perturbation upper limit ɛ should be set. Without adding the structure loss and source identity loss, we trained the model for 990 epochs under different perturbation upper limit ɛ. The ASR results for various offline face recognition models, as well as the SSIM metrics of the generated adversarial faces, are shown in Fig. 6.

As shown in Fig. 6, with the perturbation upper limit ɛ increases, the SSIM gradually decreases. Meanwhile, the values of each target model’s ASR₁ increase and converge when the perturbation upper limit ɛ = 5. However, ASR₂ shows an initial increase followed by a decrease. To investigate the reason for the decline in ASR₂, we trained the model for 990 epochs under different perturbation upper limit ɛ. The FSS and FTS data of the 6,000 adversarial faces generated by each model in various face recognition models are shown in Fig. 7.

From Fig. 7, we can see that with the increase of the perturbation upper limit ɛ, the FTS gradually rises and the FSS gradually falls. At the same time, the FTS and FSS of various face recognition models roughly intersect at the perturbation upper limit ɛ = 4, when the generated antagonistic face has the optimal “dual identity” effect, and as shown in Fig. 6, the visual metric SSIM at ɛ = 4 is also relatively high, at 94%.

In order to further visualize the superiority at ɛ = 4. We collect the FSS and FTS data of MobileFace from 6,000 adversarial faces generated under different perturbation upper limits ɛ trains 990 epoch, as shown in Fig. 8.

As shown in Fig. 8A, when ɛ = 1, the adversarial face is more likely to resemble the source identity, and when ɛ = 8, the adversarial face is more likely to resemble the target identity. From Fig. 8B, we can see that most of the adversarial faces generated when the perturbation amount is 4 are able to realize the attack effect above the threshold of “dual identity”, and realize the attack effect of resembling both the source identity and the target identity, so ɛ = 4 is the closest to the optimal solution at this time.

However, we find that adding the structure loss L_st or the source identity loss L_{adv_source} will cause the FSS to rise and the FTS to fall, and the equilibrium will be destroyed, and the experimental data when the target model is FaceNet-VGGFace2 and trains 990 epoch is shown in Table 3.

Therefore, the perturbation upper limit ɛ needs to be set to the value in Fig. 7 where FTS is greater than FSS, i.e., ɛ ∈ (4, 8]. When the perturbation upper limit ɛ = 5, by adjusting ζ = 0.92 μ = 0.15, the model can converge and balance FSS and FTS again, which 51.76&50.96. As shown in Table 3, it can be seen that both the visual quality and ASR₂ are the best. The optimized model ɛ = 5 ζ = 0.92 η = 0.15, with pixel points marked in yellow, was trained for 3,440 epochs, and after convergence, the generated samples were compared with the baseline model samples under different perturbation upper limits in the feature coordinate system of the ArcFace target model, as shown in the intuitive comparison in Fig. 9. It can also be observed that the generated adversarial samples achieved the best dual-identity effect.

Ablation of L_{adv_source} and determination of η

From the data in Table 3, we observe the following: For the model with ɛ = 5 w.o. ζ w.o. η, after adding the source identity loss with η = 0.15, the FTS decreased 2.73 But the FSS increased 9.09. For the model with ɛ = 5 ζ = 0.92 w.o. η, after adding the source identity loss with η = 0.15, the FTS decreased 2.13 But the FSS increased 6.19. The exchange ratio is close to 1:3.

Thus, the addition of the source identity loss L_{adv_source} allows us to sacrifice a small decrease in FTS for a large increase in FSS. This will leads to a small decrease in ASR₁ but a large increase in ASR₂, as well as a slight improvement in visual quality.

The tuning method for the source identity loss weight η involves gradually increasing η and observing the FSS and FTS at model convergence until they balance. As shown in the data for ɛ = 5 w.o ζ η = 0.15, the FSS and FTS are 47.82 and 53.36. Therefore, to achieve a balance between FSS and FTS for the ɛ = 5 w.o ζ model, η should be increased further from the current value of 0.15.

Ablation of L_st and determination of ζ

We collected experimental data under different structural similarity upper limits ζ with ɛ = 5 when the target model is FaceNet-VGGFace2 and trains 990 epoch, the specific data is shown in Table 4.

From the data in Table 4, we included the TruePert (the L₂ norm of the perturbation) and the Epoch (number of training epochs). The analysis reveals that after incorporating L_st, the difficulty of perturbation learning significantly increases, even with extended training epochs to 2,490, the TruePert cannot converge to ɛ. This eliminated portion corresponds to perturbation patterns that strongly affect image structure, resulting in significant improvements across all visual metrics. Regarding attack metrics, similar to the source identity loss, we observe decreased FTS and increased FSS. Notably, the ratio of FTS decline to FSS increase is only about 1:1, substantially lower than the source identity loss approximate 1:3 exchange ratio. Furthermore, when η becomes excessive, FSS may surpass FTS. Continuing to increase η or adding source identity loss would exacerbate this imbalance (FSS > FTS), leading to ASR₂ continues to decrease.

Therefore, the tuning method for the structure similarity lower limit ζ when ɛ = x is to set it to a value y larger than the SSIM of the baseline model ɛ = x, in order to remove perturbation patterns that strongly damage the image structure and enhance image quality. However, this value y cannot be too large, as it still needs to ensure that the ɛ = x ζ = y satisfy FSS < FTS, thus leaving optimization space for the source identity loss, for a more valuable exchange ratio.

In summary, the logic for determining the hyperparameters ɛ, ζ and η is as follows:

1. First, determine the perturbation upper limit ɛ which is strongly related to attack performance. Set it as a value x which FSS is less than FTS without adding the structural loss and source identity loss, from Fig. 7, x ∈ (4, 8] .

2. Then, with ɛ = x, determine the structural similarity lower limit ζ which is strongly related to visual quality. Set it as a value y which larger than the SSIM of the baseline model ɛ = x and ensuring that FSS is also smaller than FTS.

3. Finally, with ɛ = x ζ = y, determine the source identity loss weight η. Gradually increase η until FSS and FTS are balanced at convergence, which gives the final value z.

At this point, the three hyperparameters are determined, and the model trained with ɛ = x ζ = y η = z can achieve a good balance between attack performance and visual quality. As shown in Table 4, the model with ɛ = 5 ζ = 0.92 η = 0.15 reaches a balanced FSS and FTS of 52.35 and 52.23, and the ASR₂ is the highest among various parameter comparisons.

Ablation of FRs

To explain part of the source of the attack performance of the method AdvFaceGAN in this article, we conducted an ablation experiment on the ensemble model FRs using MobileFace as the black-box target model. The ablation results for the ensemble model FRs are shown in Table 5 (where the hyperparameters for each model are set as ɛ = 5 ζ = 0.92 η = 0.15). Model M₁ only uses FaceNet-VGGFace2 as the ensemble model; Model M₂ adds more loss function-related models IR50-CosFace, IR50-ArcFace, and IR50-SphereFace on top of M₁; Model M₃ adds two new structural models, Mobilenet and ShuffleNet, on top of M₂; Model M₄ is the final ensemble model in this article, which adds the new structural models ArcFace and ResNet50 on top of M₃.

By comparing the ASR of M₂ and M₁ in Table 5, it can be observed that integrating face recognition models with different loss functions provides the greatest improvement in attack effectiveness. Additionally, integrating face recognition models with different model structures also consistently enhances the attack effectiveness, although the model requires more training epochs to converge to the same perturbation amount as the complexity of the ensemble model increases.

Here, it should be mentioned that further adding face recognition models with higher recognition performance, which have different model structures or loss functions from those already integrated in the ensemble model, such as GhostFace, may lead to better attack effects, although the model convergence speed may further decrease.

Comparison experiments

Visual quality comparison experiment

To visually compare the quality of adversarial faces generated by each method, we selected five pairs of faces for a sample visual quality comparison experiment. The adversarial faces generated by each method are shown in Fig. 10. The two numbers below the face represent the feature similarity of the face to Source and Target, calculated by the Aliyun’s CompareFace API, with green representing above the 69% API threshold and red representing below the API threshold.

From Fig. 10, we can observe that gradient-based methods such as FGSM, MI-FGSM, and SiblingAttack, which is centered on the gradient method PGD, generate adversarial faces with poor quality. The perturbations are easily noticeable to the human eye. Local perturbation methods like AT3D and Adv-MakeUP, while modifying larger regions of the face, still generate noticeable adversarial patterns in local areas, which limits the visual quality of the adversarial faces and hinders their effectiveness in real-world attack scenarios. The C&W method generates adversarial faces with better visual quality; however, as we mentioned in previous experiments, C&W’s white-box attack performance is strong, but its black-box attack performance is nearly non-existent.

In contrast, adversarial faces generated by GAN-based methods such as AdvFace and AdvFaceGAN demonstrate better visual quality, highlighting the superiority of GAN-based approaches.

To quantitatively evaluate the visual quality of adversarial samples generated by different methods, we use ArcFace as the white-box substitute model to generate adversarial faces. We then calculate the attack success rate on the black-box model MobileFace. Additionally, we compute the SSIM, PSNR, and MSE metrics for the adversarial faces generated by each method to assess their visual quality quantitatively.

From the data in Table 8, it can be seen that our method, with ɛ = 5, ζ = 0.92 and η = 0.15, achieves better visual metrics than AT3D, SiblingAttack, Adv-MakeUP, and AdvFace. The SSIM of Adv-MakeUP is relatively high because it only generates makeup around the eyes in a small region, but its other two metrics are lower than ours, and its attack effectiveness is insufficient. Among traditional methods, the C&W method generates adversarial samples with unmatched visual quality, but its black-box attack transferability is very low, leading to a significant decrease in attack effectiveness. The MI-FGSM and FGSM methods slightly outperform our method in PSNR and MSE metrics, and MI-FGSM’s attack performance is also strong. However, when we use FaceNet as the white-box substitute model to generate adversarial faces and calculate the attack success rates in a black-box model (CosFace), as shown in Table 9, the transferability issues of FGSM and MI-FGSM are exposed. In contrast, our method maintains stable visual effects and attack performance, proving that it achieves the best visual quality among all the comparison methods under the same attack power and transferability conditions.

From Table 9, the insufficient transferability of FGSM and MI-FGSM attacks can be revealed, while the visual quality and attack efficacy of the proposed method remain stable. This demonstrates that our approach achieves the best visual quality among all compared methods under the same black-box impersonation attack efficacy.

Face API attack effectiveness

This section will evaluate the black-box impersonation attack effectiveness on the face comparison APIs provided by Aliyun, Tencent, and Face++. Based on previous experience with black-box attacks on offline models using three white-box models, it was found that when using ArcFace as a white-box substitute model, various gradient-based methods achieved the best black-box attack success rate for offline models. Therefore, the adversarial samples generated using different methods with ArcFace as the white-box substitute model in the previous Table 6 (offline model black-box attack experiments) will be used to evaluate these commercial APIs.

Additionally, the three face comparison APIs each provide three recommended recognition thresholds. For example, Aliyun’s recommended thresholds are [61, 69, 75], Tencent’s recommended thresholds are [40, 50, 60], and Face++’s recommended thresholds are [62.327, 69.101, 73.975]. These thresholds correspond to the recognition thresholds that achieve 0.1% FAR, 0.01% FAR, and 0.001% FAR for each commercial API. However, a higher false acceptance rate (FAR) also leads to a decrease in the true acceptance rate (TAR). Therefore, this section of the experiment first uses the recommended threshold at 0.01% FAR for all three APIs to determine whether the impersonation attack is successful. The data obtained is shown in Table 10.

By examining the data in Table 10, it can be observed that the proposed method, Our method performs slightly worse than SiblingAttack in terms of the traditional impersonation attack success rate ASR₁, but the dual-identity impersonation attack success rate is higher than all comparison methods. Additionally, the attack effectiveness remains stable across the three commercial APIs, demonstrating that the adversarial samples generated by AdvFaceGAN have excellent transferability.

Other similar studies that evaluate the success rate of attacks on commercial APIs often use the 0.1% FAR threshold, which is less secure and more vulnerable to attacks. This article also provides additional attack data at the recommended threshold for 0.1% FAR for each method, as shown in Table 11.

By examining the attack data in Table 11 at the 0.1% FAR threshold, it can be observed that the conclusions remain consistent with those at the 0.01% FAR threshold. However, it is noteworthy that the decline in ASR₂ compared to ASR₁ is less pronounced for the other methods. This is because methods that do not consider the source identity loss tend to generate adversarial faces with higher FTS but lower FSS. At a lower recognition threshold τ, the negative impact of low FSS on ASR₂ isreduced. This can be understood by referring to Eq. (12). In essence, the lower FSS weakens the detrimental effect on ASR₂at lower thresholds, which helps explain the observed differences in attack performance at varying FAR levels.

Therefore, setting a higher recognition threshold not only enhances the resistance to traditional impersonation attacks but also improves the resistance of commercial APIs to dual-identity impersonation attacks.

Dual-identity impersonation attack failure pattern diagnosis

This section of the experiment analyzes whether there are certain situations in which the adversarial samples generated by the AdvFaceGAN method in this article lead to difficulties in successful attacks. First, the distribution of the adversarial samples generated by the AdvFaceGAN method in the feature coordinate systems of three white-box models and three black-box models is plotted, as shown in Fig. 11.

As shown in Fig. 11, some samples in the top-left corner are discarded during the adversarial network optimization process and fail to achieve the dual-identity effect. These samples are visualized when attacking MobileFace in Fig. 12.

As shown in Fig. 12, it can be observed that when the source face and target face involve cross-skin tone or cross-gender situations, FTS struggles to reach the correct value, thus failing to achieve the dual-identity impersonation attack.

As shown in Fig. 11, it can be observed that the samples within the green circle in the upper right achieved the highest FSS & FTS during the optimization process of the Generative Adversarial Network, demonstrating excellent dual identity effects. The visualization of these samples when attacking MobileFace is shown in Fig. 13.

This suggests that attackers need to select victims whose gender and skin tone are similar to their own in order to improve the success rate of dual-identity impersonation attacks.

Time processing experiments

This section compares the time cost required by different attack methods for generating adversarial faces for a new source identity and target identity. The attack methods are divided into Algorithm-based and GAN-based approaches. Algorithm-based attack methods do not require pre-training of neural networks, so there is no training cost. For GAN-based attack methods, Adv-MakeUP and DiffAM need to re-train the network for a new target identity, so the total cost is the sum of training cost and inference cost. AdvFaces and the method presented in this paper, AdvFaceGAN, do not require re-training the network for a new target identity, so the total cost is the inference cost. The time is measured in seconds, and the timing for this experiment was conducted on a laptop with an i9-14900 CPU and an RTX4070 GPU configuration, Only DiffAM was trained on a cloud server with 24 GB of VRAM due to its maximum requirement of nearly 21.398 GB of VRAM.

As shown in Table 12, Algorithm-based attack methods such as C&W and SiblingAttack often require more time for multi-step iterations to achieve better attack results, making them much slower than AdvFaceGAN, which only requires inference. The training cost for GAN-based methods is faster than that of AdvFaceGAN because Adv-MakeUP and DiffAM train networks that can only generate adversarial faces for specific target identities, and AdvFaces uses FaceNet as the only one substitute model. These methods have simpler training objectives. However, AdvFaceGAN does not require re-training for new target identities, so when generating adversarial faces for new source and target identities, The total cost for AdvFaceGAN is only 0.25 s, which is fast enough.