Testing the limits: exploring adversarial techniques in AI models

Overview of adversarial techniques

Adversarial techniques exploit vulnerabilities in ML models to compromise their performance or extract sensitive information. These methods encompass various approaches to deceive or manipulate models at various stages of their lifecycle, from training to deployment. Adversarial techniques can be categorized using multiple criteria, including the adversary’s knowledge of the model (i.e., knowledge-based categories), the type of model output, or the specific goals of the attack, among others.

Based on the adversary’s knowledge of the model, adversarial techniques are commonly categorized into white-box attacks and black-box attacks (Kotyan, 2023). In white-box attacks, the adversary possesses comprehensive knowledge of the model, including its architecture and parameters. Such insight enables the precise crafting of adversarial examples to exploit the model’s vulnerabilities. A well-known technique in this context is the fast gradient sign method (FGSM), which calculates the loss gradients with respect to the input to produce malicious perturbations. In particular, in the white-box pipeline, the adversary is assumed to possess complete knowledge of the target model, including its architecture, parameters, and the ability to compute exact input gradients via backpropagation. Within this setting, the adversarial objective can be formulated as either untargeted or targeted. In the untargeted case, the adversary seeks to maximize the classification loss in order to induce any misclassification, whereas in the targeted case, the objective is to minimize the loss toward a specific target class. Additionally, all adversarial examples are restricted to the valid input domain to preserve semantic similarity with the original inputs. Given clean samples and their corresponding ground-truth labels, the preprocessing steps applied by the model, such as normalization, resizing, or data type transformations, must be accurately replicated during the attack to ensure correct gradient computation. The perturbation is then obtained by optimizing the chosen objective function using gradient-based techniques. The success of generated adversarial inputs is typically evaluated using metrics such as the accuracy for untargeted attacks, or the percentage of inputs classified into the desired target class for targeted attacks. Overall, the white-box attack pipeline represents the most powerful adversarial setting, establishing an upper bound on model vulnerability and serving as a benchmark for evaluating the robustness of machine learning systems.

By contrast, black-box attacks occur when the adversary lacks direct access to the model’s internal structure and parameters (Bountakas et al., 2023). Instead, the attacker must rely on querying the model and analyzing its outputs to infer decision boundaries. A common strategy involves training a surrogate model to approximate the target model’s behavior, allowing adversaries to develop and evaluate adversarial examples without explicit knowledge of the original model’s inner workings. More precisely, in the black-box pipeline, the adversary is assumed to have no access to the internal architecture, parameters, or gradients of the target model and can only interact with it through its outputs. Depending on the level of feedback available, black-box attacks can be categorized into three main strategies. The first is transfer-based attacks, where the adversary cannot directly query the target model but instead trains a surrogate model on data from a similar distribution. Adversarial examples are crafted on the surrogate model using white-box techniques are then transferred to the target model, relying on the transferability property of adversarial examples. The second category is score-based attacks, where the adversary can query the target model and obtain confidence scores or logits. In this setting, the attacker estimates approximate gradients by analyzing the variations in the model’s output scores when small perturbations are applied to the inputs and then uses these estimated gradients to iteratively generate adversarial examples. The third category is decision-based attacks, where only the predicted class labels are accessible. These attacks typically start from a heavily perturbed input that is already misclassified and iteratively reduce the perturbation magnitude while ensuring the example remains adversarial, using techniques such as the Boundary Attack.

While white-box access is typically unattainable in real-world settings, attackers may still obtain limited information, such as a subset of input features, output class labels, or, in the case of DNNs, intermediate representations from hidden layers. This partial insight allows adversaries to develop more informed attack strategies than in black-box settings while operating under realistic constraints. This is widely known as the gray-box attack. Here, the adversary has partial knowledge of the target model but lacks full access to its internal parameters or complete architecture. In this setting, the attacker may know aspects such as the backbone network, data distribution, preprocessing techniques, or normalization statistics, but other components, such as task-specific heads or stochastic defenses, remain unknown. Limited queries to obtain labels or confidence scores may also be permitted under strict constraints. The attack can be either untargeted, aiming to induce any misclassification, or targeted, forcing predictions into a specific class, while ensuring perturbations remain visually imperceptible and valid within the input domain. To exploit available knowledge, the adversary typically builds a calibrated surrogate model aligned with the known properties of the target system. When possible, the surrogate is refined through fine-tuning, synthetic data generation, or distillation from limited queries. For non-differentiable components or randomized defenses. Compared to white-box attacks, gray-box scenarios are more challenging but also more realistic, bridging the gap between fully transparent systems and complete black-box settings.

Adversarial attacks can also be grouped by their strategy, defining their implementation and objectives. Evasion attacks introduce small, often imperceptible perturbations to data during inference to mislead the model. Poisoning attacks compromise the training process by injecting malicious samples into the dataset. Model extraction attacks replicate a proprietary model by querying it and subsequently training a substitute. Finally, inference attacks (e.g., membership or model inference and attribute inference) aim to glean sensitive information from the model, such as whether a specific record was included in the training set.

Categories of adversarial techniques

Adversarial techniques exploit specific vulnerabilities in ML models, often targeting their behavior in various ways. While the spectrum of adversarial methods is extensive, this section concentrates on four key attack strategies—evasion attacks, poisoning attacks, model extraction attacks, and inference attacks—that exemplify adversaries’ diverse approaches to compromise models.

Evasion Attacks. Evasion attacks exploit ML models’ vulnerabilities during inference by introducing small, carefully designed perturbations to the input data. While these perturbations usually remain imperceptible to the human eye, they are crafted to deceive the model into making incorrect predictions or classifications. By targeting a model’s decision boundaries, attackers can push inputs across these boundaries with minimal modifications, thereby causing a significant degradation in the model’s performance. For instance, an adversary could subtly alter an image of a stop sign so that a computer vision model, potentially integrated into an autonomous vehicle, misclassifies it as a yield sign (Papernot et al., 2017). Such scenarios can lead to dangerous real-world consequences.

Evasion attacks fundamentally rely on gradient-based methods. Attackers compute the gradient of the model’s loss function with respect to the input data to determine the direction that maximally increases the model’s error. By applying a perturbation aligned with this direction, they create adversarial examples that appear visually unchanged to human observers. A commonly used formulation for crafting such perturbations is shown in Eq. (1) (Pantelakis et al., 2023):

(1) $$X_{adv}=x+\in \bullet sign({\mathrm{\nabla }}_{x}L(x,y))$$where $x$ is the original input, $\in$ is the perturbation magnitude, which determines the quantity of noise that is added to the input, and ${\mathrm{\nabla }}_{x}L(x,y)$ is the gradient of the loss function with respect to $x$.

The effectiveness of evasion attacks stems from deep learning models’ inherent sensitivity to slight input variations. While this sensitivity facilitates the capture of complex patterns, it also makes these models vulnerable to adversarial manipulations. A thorough understanding of evasion attacks helps researchers anticipate potential threats and develop strategies to fortify model robustness.

Poisoning attacks. Unlike evasion attacks, which target the model during inference, poisoning attacks involve intentionally manipulating training data to introduce vulnerabilities into the model. These attacks exploit the reliance of ML algorithms on clean and representative datasets by contaminating the training set with maliciously crafted samples. The adversary strategically injects these samples to influence the model’s learning process by degrading its overall performance or inducing specific erroneous behaviors under targeted conditions.

Poisoning attacks typically exploit vulnerabilities in the data collection and model training pipeline by leveraging the following mechanisms:

• Label flipping: Adversaries manipulate the labels of training samples to induce incorrect associations within the model. For example, altered labels in a facial recognition system may lead the model to misidentify individuals, thereby degrading its classification accuracy.

• Feature injection: Attackers introduce malicious examples containing irrelevant or misleading features into the training dataset. Such perturbations distort the model’s feature space and learning trajectory. For instance, inserting benign-looking but carefully crafted patterns into spam emails can cause the classifier to misinterpret or overlook legitimate spam indicators.

• Backdoor attacks: These constitute a specialized class of poisoning attacks, wherein adversaries embed specific “triggers” within the data that are covertly associated with particular target labels. During inference, the trigger compels the model to misclassify inputs, irrespective of their true content.

Poisoning attacks pose a significant threat, mainly when training data is sourced from unverified or publicly accessible origins. Such vulnerabilities are prevalent in federated learning frameworks, open-source repositories, and collaboratively curated datasets. In these contexts, attackers can insert adversarial samples into the data stream, compromising the resulting model’s integrity and reliability.

Model extraction attacks. These target ML systems by replicating their functionality through systematic querying. These attacks are particularly concerning for models deployed via publicly accessible application program interfaces (APIs), where adversaries can exploit the query-response interface to infer decision boundaries or approximate the underlying parameters of the target model. Such actions not only compromise the intellectual property of the model owner but also enable further adversarial activities, such as evasion attacks.

The tactics employed by attackers are influenced by the type of model outputs available. When soft labels (i.e., class probabilities) are accessible, adversaries can efficiently approximate model parameters with relatively few queries. In contrast, when only hard labels (i.e., predicted classes) are returned, the extraction process becomes more challenging. Nonetheless, Tramèr et al. (2016) demonstrated that linear and non-linear models can be effectively extracted even in such constrained settings. Their work showed that it is possible to replicate a target model’s behavior with high fidelity despite limited access to output information with carefully crafted queries.

Model extraction is often accompanied by surrogate training, wherein the adversary uses the collected query-response pairs to train a local model that mimics the decision-making process of the target system. This surrogate model can be exploited for unauthorized deployment, competitive advantage, or as a stepping stone for further attacks.

Mitigation strategies against model extraction attacks typically involve reducing the amount of information disclosed through APIs. This may include returning only hard labels instead of probability distributions, applying rate limiting to restrict the volume of queries, and incorporating noise or differential privacy techniques to obscure outputs. These defenses balance system usability with robust protection against unauthorized model replication.

Inference attacks. Finally, inference attacks target the privacy and confidentiality of ML systems by extracting sensitive information related to the training data, model parameters, or other private aspects. These attacks exploit the information revealed through model predictions and pose significant risks in sensitive domains such as healthcare, finance, and legal systems, where data confidentiality is paramount.

A prominent category of inference attacks is membership inference, wherein adversaries aim to determine whether a specific data point was included in the training set (Shokri et al., 2017). This is typically achieved by analyzing prediction outputs, such as confidence scores or probability distributions. Membership inference attacks are particularly concerning in contexts involving personal or medical data, as they may lead to severe privacy violations.

Another notable variant is attribute inference, where attackers attempt to deduce sensitive input data attributes, such as demographic information or behavioral characteristics, even when these attributes are not explicitly present in the dataset. Furthermore, some inference attacks extend to extracting model parameters, thereby exposing proprietary configurations and enabling unauthorized replication or exploitation.

Overfitting often facilitates these attacks, as models that memorize training data exhibit greater vulnerability to inference. Consequently, the susceptibility of ML models to inference attacks highlights the need for robust defense mechanisms. Techniques such as differential privacy, which introduces controlled noise to model outputs, can obscure sensitive information, while output restrictions, such as providing hard labels instead of confidence scores, can further mitigate information leakage. Additionally, regularization methods that reduce overfitting enhance model generalization and serve as a preventive measure against inference-based exploits.

Diving into evasion attacks

Evasion attacks are among the most prominent adversarial strategies that exploit the vulnerability of ML models to input perturbations at the inference stage. This category of attacks poses significant challenges, especially in critical applications where reliability is paramount. In this section of the works, four evasion attack techniques will be studied, including FGSM, projected gradient descent, DeepFool, and Carlini & Wagner (C&W). Each of these techniques represents a unique approach to generating adversarial examples.

Fast gradient sign method. The FGSM (Goodfellow, Shlens & Szegedy, 2014) is one of the earliest and most extensively studied adversarial attack techniques. It exploits the sensitivity of DNNs to small, deliberately crafted perturbations in input data, resulting in misclassification, while the perturbations remain imperceptible to human observers. FGSM is classified as a white-box attack, assuming the attacker fully knows the model’s architecture and parameters. The simplicity and computational efficiency of FGSM have made it a popular choice for adversarial research and benchmarking model robustness. The core idea behind FGSM is to perturb the input data in the direction of the gradient of the loss function with respect to the input. This method modifies the input so as to maximize the model’s prediction error, thereby inducing misclassification. The adversarial example $x_{\mathrm{a}\mathrm{d}\mathrm{v}}$ is computed as detailed below (see Eq. (2)) (Pantelakis et al., 2023):

(2) $$x_{\mathrm{a}\mathrm{d}\mathrm{v}}=x+\epsilon \cdot \mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\mathrm{\nabla }}_{x}L(x,y))$$where $x$ is the original input, $\epsilon$ is the perturbation magnitude controlling the amount of noise added to the input, ${\mathrm{\nabla }}_{x}L(x,y)$ denotes the gradient of the loss function L with respect to the input $x$, and $y$ is the true label associated with the input $x$.

The FGSM attack proceeds as follows: the original input $x$ is first passed through the model to compute the loss $L(x,y)$ using the true label $y$. Next, the gradient ${\mathrm{\nabla }}_{x}L(x,y)$ is calculated, indicating how changes in the input affect the loss. The sign of this gradient, $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\mathrm{\nabla }}_{x}L(x,y))$, is then used to determine the direction in which the loss increases most rapidly. $\epsilon$ scales this directional information to control the perturbation’s magnitude. Finally, the adversarial example $x_{\mathrm{a}\mathrm{d}\mathrm{v}}$ is generated by adding the scaled perturbation to the original input $x$.

Due to its effectiveness, speed, and ease of implementation, FGSM is a foundational technique in adversarial machine learning and is often employed as a baseline for evaluating the robustness of ML models.

Projected gradient descent. This is an iterative adversarial attack method designed to generate adversarial examples by optimizing perturbations within a constrained range. It is considered a more powerful and generalized extension of the FGSM. Projected gradient descent (PGD) operates by iteratively refining perturbations to maximize adversarial impact while ensuring that the modified input remains within a specified $\epsilon$-bounded region around the original input. This makes PGD particularly effective at crafting adversarial examples that remain imperceptible while successfully misleading the model.

PGD assumes full access to the target model’s architecture and parameters, and is therefore categorized as a white-box attack. The attack process involves two key steps: computing the gradient of the loss function with respect to the input and projecting the perturbed input back into the allowed $\epsilon$-ball to ensure it remains within the specified constraint.

The attack begins by adding a small random perturbation $\delta$ within the $\epsilon$-ball around the original input $x$, initializing $x_0=x+\delta$. For each iteration $t$, the gradient of the loss function $L(x_t,y)$ is computed with respect to the current input $x_t$, where $y$ is the true label. The input is then updated as follows (see Eq. (3)) (Madry et al., 2017):

(3) $$x_{t+1}={\mathrm{\Pi }}_{{\mathcal{B}}_{\epsilon }(x)}\left(x_t+\alpha \cdot \mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\mathrm{\nabla }}_{x}L(x_t,y))\right)$$where $\alpha$ is the step size controlling the magnitude of each perturbation update, ${\mathrm{\Pi }}_{{\mathcal{B}}_{\epsilon }(x)}(\cdot )$ denotes the projection operator that ensures the updated input remains within the $\epsilon$-ball ${\mathcal{B}}_{\epsilon }(x)$ centered at the original input $x$, and ${\mathrm{\nabla }}_{x}L(x_t,y)$ is the gradient of the loss function with respect to $x_t$.

After each gradient update, the projection step ensures the perturbed input stays within the allowable perturbation range. This iterative process continues for a predetermined number of steps or until the adversarial example causes the model to misclassify the input.

The key strength of PGD lies in its iterative nature, which allows for more precise perturbation refinement compared to single-step methods like FGSM. As a result, PGD can generate highly effective adversarial examples that are significantly more challenging to defend against. Due to its effectiveness and generality, PGD is often regarded as a benchmark for evaluating model robustness against adversarial attacks.

DeepFool. DeepFool is an iterative adversarial attack that seeks to find the minimal perturbation required to misclassify an input. Initially proposed by Moosavi-Dezfooli, Fawzi & Frossard (2016), the core idea involves locally linearizing the classifier’s decision boundaries at each iteration and progressively perturbing the input towards the nearest decision boundary until the model’s prediction changes. Unlike gradient-based methods such as FGSM and PGD, which apply fixed or bounded perturbations, DeepFool is designed to compute the smallest possible perturbation that causes misclassification. This often results in adversarial examples virtually imperceptible to human observers due to the minimal perturbation magnitude.

DeepFool was initially developed for binary classifiers, but its methodology can be extended to multi-class models. The attack proceeds iteratively by determining the smallest perturbation that shifts the input across the decision boundary. Let $x$ be the original input, correctly classified with the true label $y$, and denote the initial input as $x_0=x$. At each iteration $t$, the classifier’s decision boundary is approximated by linearizing the output function $f(x)$ at the current point $x_t$. For linear models, this step precisely reveals the decision boundary; for non-linear models, it yields a local linear approximation of the decision surface.

At iteration $t$, the algorithm computes the perturbation $r_t$ needed to move $x_t$ across the closest linearized decision boundary. This perturbation is calculated using Eq. (4) (Moosavi-Dezfooli, Fawzi & Frossard, 2016):

(4) $$r_t=\frac{|f(x_t)|}{|\mathrm{\nabla }f(x_t){|}^2}\cdot \mathrm{\nabla }f(x_t)$$where $f(x_t)$ represents the classifier’s output at $x_t$, and $\mathrm{\nabla }f(x_t)$ denotes the gradient of the output with respect to the input. Once $r_t$ is computed, the input is updated as $x_{t+1}=x_t+r_t$, effectively moving it closer to the decision boundary than the previous input $x_t$. This process of linear approximation, perturbation calculation, and input update is repeated iteratively until the model’s prediction for $x_t$ changes. The total perturbation required to induce misclassification is then given by $r=x_t-x$, representing the minimal adversarial adjustment necessary.

Carlini and Wagner (C&W). The adversarial attack proposed by Carlini & Wagner (2017) is regarded as one of the most effective and extensively studied methods in adversarial machine learning. This attack is distinguished by its capacity to bypass a broad range of defense mechanisms while introducing minimal, often imperceptible perturbations. The C&W attack prioritizes the generation of adversarial examples that are both misclassified by the target model and minimally altered from the original input, with perturbations typically constrained under specific norm bounds.

In contrast to simpler gradient-based methods, such as the FGSM, the C&W attack employs a more rigorous optimization framework. The core of this approach lies in formulating an objective function that jointly minimizes the perturbation magnitude and maximizes the likelihood of misclassification. Perturbation size, denoted $\delta$, is most commonly quantified using the $L_2$ norm, which measures the Euclidean distance between the original input and the perturbed sample. Alternatively, the $L_{\mathrm{\infty }}$ norm (which captures the largest absolute change across all pixels) or the $L_0$ norm (which counts the number of altered pixels, emphasizing sparsity) may be used, depending on the attack variant.

Formally, the adversarial example $x_{\mathrm{a}\mathrm{d}\mathrm{v}}$ is computed by adding a perturbation vector $r$ to the original input $x$, as shown in Eq. (5) (Carlini & Wagner, 2017):

(5) $$x_{\mathrm{a}\mathrm{d}\mathrm{v}}=x+r$$where the perturbation $r$ is constrained to ensure that its norm does not exceed a specified threshold, thereby maintaining imperceptibility. The misclassification constraint is enforced through a tailored loss function that encourages $x_{\mathrm{a}\mathrm{d}\mathrm{v}}$ to be confidently assigned to a target class (for targeted attacks) or simply misclassified (for untargeted attacks). This objective function is minimized using gradient-based optimization techniques such as Adam or SGD.

A key aspect of the C&W attack is the incorporation of input validity constraints. For instance, in image data, pixel values must lie within a valid range, typically $[0,1]$. To satisfy this, the attack introduces a change of variables, optimizing over an unconstrained parameter $w$ instead of directly optimizing $r$. The adversarial example is then obtained via a transformation involving the hyperbolic tangent function, ensuring the resulting values stay within permissible bounds, as shown in Eq. (6) (Carlini & Wagner, 2017):

(6) $$x_{\mathrm{a}\mathrm{d}\mathrm{v}}=\frac{1}{2}\cdot (\tanh (w)+1).$$

The optimization proceeds iteratively until convergence, at which point $x_{\mathrm{a}\mathrm{d}\mathrm{v}}$ represents a minimally perturbed input that causes the model to err. The C&W attack’s precision and adaptability have made it a benchmark for evaluating the robustness of ML models against adversarial threats.

Fully connected neural network

The fully connected neural network (FCNN) serves as a baseline model within the EVAISION framework. It comprises a sequence of layers that operate on flattened input vectors, with a straightforward architecture that renders it well-suited for both grayscale datasets, such as MNIST and Fashion-MNIST, and RGB datasets, such as CIFAR-10, following appropriate preprocessing. Specifically, this model employs the ReLU activation function in its hidden layers and utilizes the cross-entropy loss function during training.

In the FCNN, each input image is first transformed into a one-dimensional vector before propagating through three fully connected layers. ReLU activations are applied between layers to introduce non-linearity and facilitate learning complex representations. Due to the network’s requirement for flattened inputs, datasets undergo preprocessing as follows: $(i)$ MNIST and Fashion-MNIST images are directly reshaped from their original 2D format into 1D vectors and $(ii)$ CIFAR-10 images, originally in RGB format, are first converted to grayscale to conform to the model’s expected input structure, after which they are flattened.

The architecture of the FCNN used in this work is composed of the following layers:

1. The first fully connected layer (FC1) maps the flattened input vector to 128 neurons.

2. The second layer (FC2) further reduces the dimensionality by mapping to 64 neurons.

3. The final output layer (FC3) produces 10 outputs, each corresponding to one of the 10 classes present in the MNIST dataset.

LeNet

LeNet is a classical CNN architecture supported by EVAISION for assessing adversarial attacks. It comprises a series of convolutional layers followed by max-pooling operations, which progressively reduce spatial dimensions, culminating in fully connected layers for classification. The model employs the ReLU activation function and is optimized using the Adam optimizer to facilitate efficient and stable convergence during training.

Initially designed for grayscale image datasets such as MNIST and Fashion-MNIST, LeNet has been adapted in this work to support the CIFAR-10 dataset through appropriate preprocessing steps prior to input into the network.

The LeNet architecture used in this study comprises two convolutional layers and three fully connected layers. Specifically, for the convolutional layers: $(i)$ Conv1—Applies 6 filters of size $5\times 5$ with padding to maintain the original spatial dimensions and $(ii)$ Conv2—Applies 16 filters of size $5\times 5$ to capture more complex features. Next, for the fully connected layers: $(i)$ FC1—Transforms the flattened output from the convolutional layers into a 120-dimensional feature vector, $(ii)$ FC2—Reduces the dimensionality to 84 neurons, and $(iii)$ FC3—Outputs 10 neurons, corresponding to the number of target classes in MNIST.

The convolutional layers extract both low-level and high-level features from the input images. Max-pooling is applied after each convolutional layer to downsample feature maps and reduce computational complexity. The fully connected layers integrate these features and perform the final classification.

MobileNetV2

MobileNetV2 utilizes two convolutional layers to extract low- and high-level features from the images, while a maximum grouping layer helps reduce spatial dimensions after the second convolution. Its fully connected layers process the extracted features for final classification. In this article, MobileNetV2 is loaded using the Torchvision library (PyTorch Foundation, 2025).

Given that MobileNetV2 is designed to process three-channel images, preprocessing steps were applied to ensure compatibility across all datasets. Input adjustment: Grayscale images from MNIST and Fashion MNIST were replicated across three channels to conform to the expected input format. Output adjustment: The original classification head was replaced with a custom fully connected layer configured to output predictions for 10 classes, corresponding to digits (MNIST), clothing categories (Fashion-MNIST), or object classes (CIFAR-10).

VGG11

VGG11 is a well-established CNN architecture characterized by a sequential arrangement of convolutional layers followed by fully connected layers. Its structured and deep design has proven effective in tasks requiring robustness analysis, such as adversarial evaluation. In this work, VGG11 was also loaded from the Torchvision library.

To adapt VGG11 for use with all datasets in this study, the following modifications were implemented. Input layer adjustment: The first convolutional layer was modified to accept single-channel grayscale images for the MNIST and Fashion MNIST datasets. Input data adjustment: All images were resized to $224\times 224$ pixels to match VGG11’s expected input dimensions. Output adjustment: The final fully connected layer was replaced with a new layer configured to output predictions for 10 classes, corresponding to the labels in MNIST, Fashion-MNIST, and CIFAR-10.