Neural noiseprint transfer: a generic noiseprint-based counter forensics framework

Deep feature representations

We studied the deep feature representations of the CNN denoiser (Zhang et al., 2017) in a similar way that was used before toward the goal of understanding the deep image representations (Mahendran & Vedaldi, 2015; Lin & Maji, 2016) and for artistic style transfer approaches (Gatys, Ecker & Bethge, 2016; Johnson, Alahi & Fei-Fei, 2016). Specifically, we present an input image I to the CNN denoiser and produce the features at a specific convolutional layer l. Then, we find another image J that its features at the same convolutional layer best match the features produced for I. Mathematically, (1)${\displaystyle \mathbf{J}=arg\underset{\mathbf{X}}{min}{\parallel Fl(I)\; -\; Fl(X)\parallel }_2^2,}$ where ${\mathcal{F}}^l\left(\mathbf{X}\right)$ is the features produced for image X at the convolutional layer l.

Figure 4 shows the produced images obtained from the features of different convolutional layers of the CNN denoiser using Eq. (1). As shown in the figure, the images produced from the features of the early layers tend to preserve the content of the input image. However, as we go deeper in the network, the produced images tend to lose this content fidelity with the input image and start to show high similarity with the noiseprint of the input image. Thus, the features at the early and end layers can provide content and a noiseprint representations, respectively, of an image in the feature space.

These content and noiseprint representations are employed in both the optimization-based approach and the training of the noiseprint-injector. Specifically, we utilize loss functions that penalize both approaches’ differences in content and noiseprint representations in the feature space. We choose the L2 loss because it is a convex and continuously-differentiable function. This allows the gradient-based optimization to find the global minimum. Please note that the original noiseprint extraction method (Cozzolino & Verdoliva, 2020) employed cross-entropy loss for the output of the binary classifier as shown in Fig. 2. This is because the goal is to encourage the Siamese architecture to generate a similar output for positive class patches and a different output for negative class patches. However, this is different from the goal of our loss which penalizes the differences in content and noiseprint deep representations. The following two subsections present the two proposed approaches in detail.

Proposed optimization based neural noiseprint transfer approach

The proposed optimization-based approach is sketched in Fig. 5. The approach estimates the generated image I_g as (2)${\displaystyle {\mathbf{I}}_{\text{g}}=arg\underset{\mathbf{I}}{min}\ell \left({\mathbf{I}}_{\text{f}},{\mathbf{I}}_{\text{a}},\mathbf{I}\right),}$ where (3)${\displaystyle \ell \left(\mathbf{X},\mathbf{Y},\mathbf{Z}\right)=\alpha {\mathcal{L}}_{\text{c}}\left(\mathbf{X},\mathbf{Z}\right)+\beta {\mathcal{L}}_n\left(\mathbf{Y},\mathbf{Z}\right)+\gamma {\mathcal{L}}_p\left(\mathbf{X},\mathbf{Z}\right)}$ is a loss function that comprises the following terms:

• The content loss ${\mathcal{L}}_{\text{c}}\left(\mathbf{X},\mathbf{Z}\right)={\parallel Fc(X)\; -\; Fc(Z)\parallel }_2^2$ that penalizes the difference in content representations of X and Z at convolutional layer c.

• The noiseprint loss ${\mathcal{L}}_n\left(\mathbf{Y},\mathbf{Z}\right)={\parallel Fn(Y\; )\; -\; Fn(Z)\parallel }_2^2$ that penalizes the difference in noiseprint representations of Y and Z at convolutional layer n.

• The pixel loss ${\mathcal{L}}_p\left(\mathbf{X},\mathbf{Z}\right)={\parallel X\; -\; Z\parallel }_2^2$ that penalizes the difference in pixel domain between X and Z.

The factors α, β, and γ are weight-balancing factors between the different loss terms.

To solve Eq. (2), several numerical optimization techniques such as stochastic gradient descent or ADAM (Kingma & Ba, 2015) can be used, but our experiments show that the latter produces better results. In this case, we need to compute ∇ℓ, which is the gradient of the loss function ℓ w.r.t. I. Note that the gradient of ℓ involves computing the differentiation of the deep feature representations obtained from different levels of the original noiseprint extraction network (Cozzolino & Verdoliva, 2020) as shown in Fig. 5. Luckily, these representations are typically composed of a chain of several linear and non-linear layers and are still differentiable. Additionally, since the optimization here is performed solely for the generated image (not the weights of the network), the network weights are kept fixed with the pre-trained weights from the original noiseprint extraction network. The detailed implementation of the proposed optimization-based approach is sketched in Algorithm 1.

Finally, as we solve Eq. (2) to estimate an image, the optimization is time-consuming as it involves a large number of computations, especially for large-size images. As we detail next, this problem is mitigated in our other proposed injection-based approach.

 
________________________________________________________________________________ 
Algorithm 1: Proposed optimization-based approach. 
________________________________________________________________________________ 
  Data:   If  and Ia 
  Result:   Ig 
1 Na ←Fn(Ia)  /* get noiseprint representation of Ia*/  ; 
2 Cf ←Fc(If)  /* get content representation of If*/  ; 
3 I ← 0; 
4 Epoch ← 0; 
5 while Epoch is less than the total required number of epochs  do 
6   Ni ←Fn(I)  /* get noiseprint representation of I*/  ; 
7   Ci ←Fc(I)  /* get content representation of I*/  ; 
8   Lc ← α∥Cf − Ci∥22; 
9   Ln ← β∥Na − Ni∥22; 
10  Lp ← γ∥If − I∥22; 
11  L←Lc + Ln + Lp; 
12  Perform one update step for I using the ADAM optimizer by minimizing L 
     w.r.t.  I; 
13  epoch ← epoch + 1; 
14 Ig ← I; 
________________________________________________________________________________    

Proposed injection based neural noiseprint transfer approach

Instead of optimizing for the generated image, which is time-consuming for large images, as we presented in the previous subsection, here we propose another much faster approach. Specifically, we propose a novel noiseprint-injector neural network that learns how to transfer the noiseprint of an image into another one. The training of the network is performed offline and completely in an unsupervised fashion. Let the noiseprint-injector network be denoted as $\mathcal{G}$ and parameterized by the parameter vector w. As shown in Fig. 6A, the noiseprint-injector is trained to take the content and noiseprint representations of two different images X and Y, and produce another image that has:

• Similar content representations in the feature space with the first image X.

• Similar values in pixel domain with X.

• Similar noiseprint representations in the feature space of the second image Y.

In other words, the goal of training the noiseprint-injector network is to make the network learns how to produce an output image that bears the noiseprint of the image Y while looking visually similar to X. This goal can be achieved by estimating the best parameter vector w^∗ of the noiseprint-injector network that minimizes the loss ℓ defined in Eq. (3). Mathematically, (4)${\displaystyle {\mathbf{w}}^{\ast }=arg\underset{\mathbf{w}}{min}\sum _{\forall \mathbf{X}\ne \mathbf{Y}}\ell \left(\mathbf{X},\mathbf{Y},\mathcal{G}\left({\mathcal{F}}^c\left(\mathbf{X}\right),{\mathcal{F}}^n\left(\mathbf{Y}\right),\mathbf{w}\right)\right).}$ The training procedure of the proposed noiseprint-injector network (shown in Fig. 6A) comprises the following steps.¹ At each learning epoch, we first get the content representation of X and the noiseprint representation of Y. Then we perform a forward pass of the noiseprint-injector network to get the output image. Then, we obtain both the content and the noiseprint representations of the output image and form the total loss in Eq. (3) with these representations in addition to the representations extracted from X and Y. Finally, we perform a backward optimization step for the noiseprint-injector network using the ADAM optimizer to update the parameters w. The detailed implementation of these steps is presented in Algorithm 2. After estimating w^∗, the noiseprint-injector can produce the generated image from the authentic and forged images as (5)${\displaystyle {\mathbf{I}}_{\text{g}}=\mathcal{G}\left({\mathcal{F}}^c\left({\mathbf{I}}_{\text{f}}\right),{\mathcal{F}}^n\left({\mathbf{I}}_{\text{a}}\right),{\mathbf{w}}^{\ast }\right).}$ Specifically, we get the content representation of I_f and the noiseprint representation of I_a and feed them to the noiseprint-injector network (with its weights fixed to w^∗) to produce I_g.

The architecture of the proposed noiseprint-injector network is shown in Fig. 6B. The details of the input–output sizes, kernel, and stride lengths of each layer are shown in Table 1. First, we combine the input content and noise representations in the feature space by stacking them.² Because the content and noise representations differ in the dynamic range, we multiply the noise representations by a factor λ before combing them. Then, the stacked features are presented to a convolution deconvolution auto-encoder network with symmetric skip connections. A ReLU activation layer follows each deconvolution layer. Our noiseprint-injector network is similar to the network proposed in Mao, Shen & Yang (2016) for image restoration. However, our network here is much shallower, and the inputs to the network are feature space representations, not an image as in Mao, Shen & Yang (2016).

We emphasize that the goal of training our noiseprint-injector neural network is to learn how to transfer the noiseprint of an image into another one. There is no restriction about the image-pair used for the training. The only requirement of the image pair is to contain images with different noiseprints so the network can learn how to transfer the noiseprint from one image into another while generating a high-fidelity image. This requirement of different noiseprints can be easily fulfilled by just using images obtained from different cameras or from the same camera but cropped at different spatial locations. This again enforces our claim that the proposed noiseprint-injection approach is generic and does not require any sophisticated datasets (that are not easy to obtain) in training. After training, the generated image is estimated in one forward pass of the network without any optimization. Therefore, the proposed noiseprint injection-based approach is much faster than the proposed optimization-based approach discussed in the previous subsection. Also, as the proposed architecture does not include any size-changing layer (such as pooling), the architecture works for any input size (a × b). In other words, there is no need to split the input image into non-overlapping windows with specific sizes to make it suitable for the network.

 
________________________________________________________________________________________ 
 Algorithm 2: Illustration of the proposed noiseprint-injector network 
  training procedure. 
_______________________________________________________________________________________ 
    Data:   Dataset that contains images with different noiseprints 
    Result:   Best parameter vector w∗ 
 1  Epoch ← 0; 
 2  Initialize the parameter vector w; 
 3  while Epoch is less than the total required number of epochs  do 
 4    Get two different images X and Y from the dataset; 
 5    Cx ←Fc(X)  /* get content representation of X*/  ; 
 6    Ny ←Fn(Y)  /* get noiseprint representation of Y*/  ; 
 7    I ←G(Cx,Ny,w)  /* forward pass to generate an output from the proposed 
       noiseprint-injector network */  ; 
 8    Ci ←Fc(I)  /* get content representation of I*/  ; 
 9    Ni ←Fn(I)  /* get noiseprint representation of I*/  ; 
 10   Lc ← α∥Cx − Ci∥22; 
 11   Ln ← β∥Ny − Ni∥22; 
 12   Lp ← γ∥X − I∥22; 
 13   L←Lc + Ln + Lp; 
 14   Perform one update step for w using the ADAM optimizer by minimizing L 
       w.r.t.  w (backward pass); 
 15   epoch ← epoch + 1; 
 16 w∗ ← w; 
______________________________________________________________________________________    

Training and parameters setting

Forgery localization attack

We evaluate the effectiveness of the proposed counter-forensic approaches in attacking the OrgNoiseprint and the Comprint forgery localization methods.³ Additionally, the performance of the proposed approaches is compared with the median filtering-based approach. We used both 3 × 3 and 5 × 5 kernel sizes for the median based-approaches. All counter-forensic approaches under comparison (the proposed and the median filter approaches) are applied to forged images to produce newly generated images. Then, the forgery is localized in the generated images by searching for inconsistencies in their noiseprints using the Splicebuster blind localization algorithm (Cozzolino, Poggi & Verdoliva, 2015). The output of the localization algorithm is a real-valued heat map that indicates, for each pixel, the likelihood that it has been forged.

To provide a quantitative evaluation of the counter-forensic approaches under comparison, we evaluate them in two aspects. First, we need to measure the degree of visual similarity between the forged images and the generated ones. The PSNR and the SSIM (Wang et al., 2004) measures are used for this purpose. Second, we treat forgery localization as a binary classification problem in which pixels of the generated image can either belong to forged or authentic classes. Then, we employ several measures to evaluate the effect of the approaches under comparison on the performance of this binary classification results. All employed measures are built on four fundamental quantities, which are: (a) TP (true positive): # forged pixels declared forged, TN (true negative): # authentic pixels declared authentic, (c) FP (false positive): # authentic pixels declared forged, and (d) FN (false negative): # forged pixels declared authentic. From these four quantities, we measure the F1 score, Matthews Correlation Coefficient (MCC), and average precision (AP) as in Cozzolino & Verdoliva (2020). Note that, as in Cozzolino & Verdoliva (2020), the heat map generated by the localization algorithm is thresholded. Then, the F1 and MCC scores are reported as the maximum over all possible thresholds. Also, the AP is computed as the area under the precision–recall curve. Three datasets are used in our evaluation of the counter forgery localization, which are the DSO-1 dataset (de Carvalho et al., 2013), Decision-Fusion dataset (Fontani et al., 2013), and the Korus dataset (Korus & Huang, 2017). Each dataset has different characteristics. The DSO-1 dataset is characterized by large splicings and uncompressed images. The Korus dataset contains raw images (not JPEG compressed) with several forgeries, such as copy-move and object removals. The Decision-Fusion dataset mimics a realistic scenario of professional forgery and saving the forged images in JPEG compressed format. Note that the input to the proposed approaches is a forged image from a dataset with a corresponding same-size authentic image selected randomly. In contrast, the median filtering approach’s input is only a forged image.

Tables 2 and 3 show the quantitative evaluation of the counter-forensic approaches under comparison in attacking the OrgNoiseprint and the Comprint forgery localization methods, respectively. Specifically, for each dataset, the average F1 score, MCC score, and AP values are reported for the proposed optimization-based, the proposed noiseprint injection-based, and the median filtering-based approaches. We also report the average PSNR and SSIM between the generated images and the corresponding forged ones. To verify the consistency of the performance of the proposed approaches w.r.t. the random selection of the authentic images, the experiment is repeated ten times for each forged image. As shown in the tables, the proposed approaches significantly reduce the values of the used metrics (F1, MCC, and AP) compared with the original values associated with the OrgNoiseprint and the Comprint forgery localization methods. Also, the proposed approaches produce better metrics values compared with the median filtering-based approach. However, the median based-approach with 5 × 5 kernel produces better results than the one with 3 × 3 kernel but at the expense of more degradation to the visual fidelity. At the same time, the generated images by the proposed approaches show high visual fidelity to the forged ones, as reported by the PSNR and SSIM average values. We can see that the proposed approaches reduce the forensic accuracy scores for the DSO-1 dataset by an average of 75% compared with the OrgNoiseprint method while at the same time keeping an average PSNR of 31.5 dB and SSIM of 0.9.

The numerical results shown in the tables are reinforced by presenting some visual results of the OrgNoiseprint approach, the proposed optimization-based approach, and the proposed injection-based approach in Figs. 7, 8, and 9, respectively. In each row in Fig. 7, the columns from left to the right show the forged image, its extracted noiseprint, and its heat map. Similarly, in each row in Figs. 8 and 9, the columns from left to the right show the generated image, the noiseprint of the generated image, and its heat map. Below each row, we report the values of the F1 score, MCC score, and AP value. Again, the reported values show the effectiveness of the proposed approaches, which significantly reduces the values of the used metrics (F1, MCC, and AP) compared with the original values associated with the OrgNoiseprint method.

Please note that despite the median filtering based-approach producing fair results, the approach is vulnerable to easy detection (Kirchner & Fridrich, 2010; Cao et al., 2010; Zhu, Gu & Chen, 2022). Based on the median filtering characteristics, several hand-crafted features are designed and used along with machine learning algorithms for the detection of the application of the median filter on a digital image. Examples of the employed features from the filtered image include the gradient between neighboring pixels, the Local Binary Patterns (Zhu, Gu & Chen, 2022), the Fourier Transform coefficients (Rhee, 2015), and the singular values (Amanipour & Ghaemmaghami, 2019). In contrast, the proposed approaches synthesize a newly generated image that is visually imperceptible to a forged image, but its noiseprint is very close to the noiseprint of an authentic image. Thus, the proposed approaches do not leave statistical traces that can be detected, as in the case of the median filtering-based approach.

A final note we want to highlight is that the proposed approaches are designed to attack the forensics approaches that are built on the utilization of the noiseprint. So, the performance of the proposed approaches is closely tied to the success of the noiseprint-based forensics approaches. If the original noiseprint-based forensics approaches do not detect a specific type of forgery, the proposed approaches will not succeed accordingly. This is clear from the reported results in Tables 2 and 3 for the Korus dataset. The original noiseprint-based forensics approaches do not behave well for this dataset because the dataset constitutes raw images (not JPEG compressed) while noiseprintis trained on JPEG-compressed images and also because the dataset contains several contemporary forgeries.

Camera model identification attack

Here, we evaluate the effect of the proposed approaches and the median filter approach on the camera model identification application. Specifically, the IEEE Forensic Camera Model Identification Challenge hosted on the Kaggle platform (https://www.kaggle.com/competitions/sp-society-camera-model-identification) is used for this purpose. Five camera models are used from the dataset: Sony-NEX-7, HTC-1-M7, iPhone-6, LG-Nexus-5x, and Samsung-Galaxy-Note 3. As in Cozzolino & Verdoliva (2020), 100 training images are used from each camera to estimate the noiseprint of the camera by averaging the noiseprint extracted from each training image. Then, we crop the central 500 × 500 of 100 different test images from each camera to be used in the identification. A test image is related to a camera if the Euclidean distance from its noiseprint to the average noiseprint of the camera is minimum.

As we discussed, the input to our proposed approaches is an image pair. The approaches produce a generated image visually similar to the first input image while having similar noiseprint with the second one. Thus, to apply the proposed approaches in this attack, we form several input image pairs. In each pair, the first image is a test image obtained from a specific camera in the dataset, while the second is taken at random from the test images of a different camera.

Figure 10 shows the confusion matrices obtained for the camera model identification using: (a) the original noiseprint forensics method (Cozzolino & Verdoliva, 2020), (b) after applying the proposed optimization-based approach, (c) after applying the proposed noiseprint-injection based approach, and (d) after applying the median-filter based approach. As shown in the figure, the proposed approaches can significantly reduce the accuracy of the confusion matrix, which means that the proposed approaches are very effective in camera model identification attacks.

Discussion

In this section, we present several discussion points about the proposed approaches. First, the execution time of the proposed approaches is discussed. Then, the success of the proposed approaches in transferring the noiseprint of the authentic images to the generated ones is highlighted. Then, the consistency of the performance of the proposed approaches is verified against the different selections of authentic input images. Then, we highlight the similarity between the proposed and other existing approaches dedicated to artistic style transfer. Finally, some future research directions are presented.

Noiseprint transfer

To show the success of the proposed approaches in transferring the noiseprint from the authentic image to the generated one, we performed the same experiment in Table 2 and recorded the mean squared error of the noiseprint between the generated image and (a) the forged image (Generated vs. Forged) and (b) the authentic image (Generated vs. Authentic). Since we repeated the experiments in Table 2 ten times for each forged image, we used the box plot to represent the obtained (Generated vs. Forged) and (Generated vs. Authentic) values. These plots are shown for the first ten images⁴ from the DSO-1 and Korus datasets in Figs. 11A and 11B, respectively. Also, a solid line is drawn in each figure to connect the average value of each box for better visualization. The upper and lower rows of the figures display the results for the optimization and the injection-based approaches, respectively.

As shown in the figure, the reported (Generated vs. Authentic) values are significantly lower than the (Generated vs. Forged) values. In other words, the noiseprint similarity between the generated and authentic images is significant compared with the noiseprint similarity between the generated images and the original forged ones. This means that the proposed approaches successfully transfer the noiseprint from the authentic image to the forged one.

Performance consistency

Since the input to the proposed approaches is a forged image from a dataset with a corresponding same-size authentic image selected randomly, we need to verify the consistency of the performance with the different selections of authentic images. In Tables 2 and 3, we repeated the experiments ten times for each forged image and reported the average values. However, the average value does not indicate the spread of the reported numbers. Here, instead of reporting the average values only, we used the box plot to represent the obtained F1, MCC, and AP values for the proposed approaches for the first 10 images from the DSO-1 dataset. We show these box plots in Fig. 12.

From the box plots, we can see that the proposed approaches produce consistent (F1, MCC, AP) values regardless of the used authentic images, as shown from the spread of each box of the ten images. This is because the approaches successfully transfer the noiseprint from the authentic image to the generated one, as shown in the previous subsection. This successful transfer makes the noiseprint of the generated images not contain any traces of inconsistencies as an indication of forgery. Thus, the forgery can not be localized regardless of the authentic image used in the transfer, resulting in consistent metric values.