Multilayered digital image encryption approach to resist cryptographic attacks for cybersecurity

Advanced encryption standard

It is a symmetric block cipher widely used for the encryption of data. It operates on fixed 128-bit blocks and supports key lengths of 128, 192, or 256 bits of National Institute of Standards and Technology et al. (2001), Heron (2009). The AES encryption process consists of several transformation steps applied to the data block. First, Key Expansion generates a set of round keys from the original encryption key using a predefined key schedule. Then, the SubBytes step introduces nonlinearity by substituting each byte in the state matrix with a corresponding value from the Rijndael S-box. This is followed by ShiftRows, which performs a cyclic shift of the rows in the state matrix to further diffuse the input. Next, MixColumns transforms each column using a linear mixing function to increase diffusion across the state. Finally, the AddRoundKey step combines the current state with the corresponding round key using a bitwise XOR operation. These steps are repeated over multiple rounds depending on the key size to ensure strong encryption.

The transformation at round $r$ is:

(1) $$S_r=\mathrm{M}\mathrm{i}\mathrm{x}\mathrm{C}\mathrm{o}\mathrm{l}\mathrm{u}\mathrm{m}\mathrm{n}\mathrm{s}(\mathrm{S}\mathrm{h}\mathrm{i}\mathrm{f}\mathrm{t}\mathrm{R}\mathrm{o}\mathrm{w}\mathrm{s}(\mathrm{S}\mathrm{u}\mathrm{b}\mathrm{B}\mathrm{y}\mathrm{t}\mathrm{e}\mathrm{s}(S_{r-1})))\oplus K_r.$$In the proposed scheme, AES is employed in three different operating modes to improve security and make it suitable for images. The CBC mode is used to implement layered encryption by introducing chained dependencies between blocks, making each ciphertext block dependent on all preceding blocks. The CFB mode enables the encryption of streaming data, such as files, by converting a block cipher into a self-synchronizing stream cipher. Finally, the GCM is utilized to provide both confidentiality and data authenticity, combining encryption with a built-in authentication mechanism.

Triple data encryption standard

Improves DES security by applying it three times using multiple keys (Sari, Rachmawanto & Haryanto, 2018). Given three keys $3D{K}_1$, $3D{K}_2$, and $3D{K}_3$, encryption is defined as:

(2) $$CT=E_{3D{K}_3}(D_{3D{K}_2}(E_{3D{K}_1}(PT)))$$where PT is the plaintext and CT is the ciphertext.

In our model, 3DES is used in the CBC mode as an additional encryption layer after AES, reinforcing diffusion and resistance against differential attacks.

Elliptic curve cryptography

ECC is a public key cryptographic method based on elliptic curves in finite fields (Hankerson & Menezes, 2021). Provides strong security with relatively short key sizes, which is ideal for lightweight applications. The general curve is defined as:

(3) $$v^2=u^3+\alpha u+\beta \mathrm{m}\mathrm{o}\mathrm{d}p$$where $\alpha$ and $\beta$ define the elliptic curve and $p$ is a prime modulus.

ECC is employed here using the ECDH key exchange:

(4) $$Q_A=d_{A}G,Q_B=d_{B}G,K_{AB}=d_A{Q}_B=d_B{Q}_A$$where G is a generator point, $d_A$ and $d_B$ are private keys, and $Q_A$, $Q_B$ are corresponding public keys.

HMAC-based key derivation function

HKDF is used to derive strong session keys from shared secrets (e.g., ECC-derived keys) using HMAC (Krawczyk & Eronen, 2010). It ensures randomness and context specificity. It operates in two steps:

Extract:

(5) $$PRK={\mathrm{H}\mathrm{M}\mathrm{A}\mathrm{C}}_s(IKM)$$where IKM is the input keying material and $s$ is an optional salt.

Expand:

(6) $$OKM={\mathrm{H}\mathrm{M}\mathrm{A}\mathrm{C}}_{PRK}(info\parallel \mathrm{c}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r})$$where $info$ is optional context-specific information, and OKM is the output key material.

In this system, HKDF derives a symmetric AES key used to encrypt metadata and session parameters.

Pixel shuffling

Pixel shuffling is a spatial transformation that improves confusion by permuting pixel positions using a pseudorandom permutation $\pi$ (Bansal et al., 2017). The mapping is given by:

(7) $$J(m,n)=F(\pi (m,n))$$where F denotes the original image and J represents the shuffled result.

A secure random seed ensures reproducibility for decryption. This step masks spatial features prior to byte-level processing.

XNOR operation

The XNOR operation is a bitwise logical function that enhances diffusion by introducing non-linear transformations at the binary level. It can be calculated as:

(8) $$X1\odot X_2=\mathrm{\neg }(X_1\oplus X_2)$$where $X_1$ and $X_2$ are bit sequences, $\oplus$ represents the XOR operation, and $\mathrm{N}\mathrm{O}\mathrm{T}$ denotes the logical negation.

A binary mask derived from a secure seed is used to XNOR the image byte stream, adding an extra layer of obfuscation prior to encryption.

Encryption steps

The encryption procedure in the proposed scheme involves a series of transformations to ensure a high level of security. It includes, pixel shuffling, bitwise operations, symmetric encryption, and ECC. The Algorithm 1 presents the overall steps in detail. Initially, the input image is loaded, reshaped into a 1D array using NumPy and subjected to pixel shuffling using a pseudorandom seed. It obfuscates the spatial relationship between pixels. This shuffled image is subsequently transformed into a byte stream for further processing.

Next, a bitwise XNOR operation is applied. A binary mask of the same size as the byte stream is generated randomly. Each byte is masked using the XNOR operation. This step significantly increases confusion in the encrypted image data.

After masking, symmetric encryption keys are generated, including three 128-bit keys for AES and two 168-bit keys for 3DES. The byte stream is then encrypted in five sequential layers. The first three layers use AES in CBC mode with each of the three AES keys, followed by two layers of 3DES encryption in CBC mode using the 3DES keys. Each encryption layer employs a unique Initialization Vector (IV), which is saved for decryption.

The output of the final encryption layer is converted into a 2D grayscale image and saved as the encrypted image. All encryption metadata, including AES and 3DES keys, IVs, pixel shuffle indices, XNOR mask, and random seed, are compressed into a secure archive named encryption_info.zip.

To further secure the encryption metadata, a 256-bit AES key is randomly generated and used to encrypt the encryption_info.zip file in the CFB mode of AES. For secure key exchange, an ECC key pair is generated. Using the recipient’s public ECC key, a shared secret is calculated using the ECDH key agreement protocol. A strong symmetric key is derived from this shared secret using the HKDF, which is then used to encrypt the 256-bit AES key using AES in GCM. The final package, named final_encrypted_files.zip, contains the encrypted ZIP archive, the encrypted AES key, and the ephemeral ECC public key.

Decryption steps

The decryption process reverses all the operations performed during encryption to restore the original image. The whole decryption process can be visualized in the Algorithm 2. First, the archive final_encrypted_files.zip is extracted to retrieve the encrypted metadata file (encrypted_my_files.zip), the encrypted AES key, and the ephemeral ECC public key of the sender. Using the private ECC key of the receiver and the public key of the sender, a shared secret is regenerated through the ECDH protocol. This shared secret is passed through HKDF to derive the AES key required to decrypt the encrypted AES file key using AES-GCM.

With the decrypted 256-bit AES key, the system proceeds to decrypt encrypted_my_files.zip using AES in CFB mode, thus obtaining the encryption_info.zip. Once extracted, this archive provides all necessary decryption metadata, including keys, IVs, XNOR mask, and shuffle parameters.

The encrypted grayscale image is then converted back into a byte stream and decrypted through the five symmetric layers in reverse order. Specifically, the data are first decrypted using 3DES with the second key, followed by the first 3DES key, and then using AES with the third, second, and first AES keys, respectively, all in CBC mode. The appropriate IVs are used for each decryption stage accordingly.

After layered decryption, the bitwise XNOR operation is reversed using the saved mask to retrieve the original masked byte stream. This stream is converted back into the RGB image format and inverse pixel shuffling is applied using the previously saved random seed and index order. The reconstructed image is then utilized further.

System specifications

The hardware and software configurations for the implementation of the proposed model are mentioned in Table 3.

Experimental setup

To evaluate the effectiveness of our proposed method, we performed experiments using the USC-SIPI Image Database. It is a comprehensive collection of digitized images that supports research in image processing and machine vision. Established in 1977 and continuously updated, the database is organized into volumes, with images categorized on the basis of their characteristics. Black-and-white images are stored at 8 bits/pixel, and color images at 24 bits/pixel (Signal and Image Processing Institute (SIPI), 2025). Table 4 summarizes the dataset and sample images from different categories are shown in Fig. 2.

Validation metrics

An effective and robust encryption strategy consistently delivers strong performance on various quality assessment metrics. Figure 3 illustrates several metrics widely used to evaluate the security of image encryption techniques (Rout & Mishra, 2024; Pati, Mishra & Rout, 2024; Kaur & Kumar, 2020).

Keyspace analysis

The keyspace is defined as the total number of possible keys, which directly influences the system’s strength against brute-force attacks. A larger keyspace exponentially increases the computational complexity of key guessing, making unauthorized decryption practically infeasible. The proposed encryption algorithm employs multiple keys derived from symmetric, asymmetric, and chaotic components. The specifications of the key used in the proposed encryption scheme are given in Table 5. Assuming all keys are independently and uniformly distributed, the total keyspace is the product of all individual keyspaces, calculated as:

$$\begin{array}{rl}\mathrm{K}\mathrm{e}\mathrm{y}\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{c}\mathrm{e}& =2^{128}\times 2^{128}\times 2^{128}\times 2^{168}\times 2^{168}\times 2^{8L}\times 2^{384}\times 2^{256}\\ & =2^{(128+128+128+168+168+8L+384+256)}\\ & =2^{(1,560+8L)}\end{array}.$$

Here, L denotes the length in bytes of the image data subjected to the XNOR masking step. For typical image sizes (for example, a 100 KB image), the term $8L$ contributes approximately 800,000 bits of entropy, which vastly dominates the total keyspace.

This immense keyspace ensures that exhaustive key search attacks are computationally infeasible, even with state-of-the-art hardware. Thus, the proposed encryption framework provides a robust security guarantee, effectively mitigating brute-force and key-recovery attacks.

Key sensitivity

It refers to the ability of a cryptographic algorithm to produce significantly different ciphertexts when the encryption key is slightly modified, such as by flipping a single bit. This ensures high non-linearity and unpredictability between plaintext, key, and ciphertext, thereby enhancing resistance to cryptanalytic attacks. The Avalanche Effect (AE) quantifies key sensitivity, where an ideal algorithm exhibits an AE value greater than 50%, which means that a one-bit change in the key alters approximately half of the ciphertext bits (Kaur & Kumar, 2022). This property strengthens diffusion, making it infeasible for attackers to infer relationships among plaintext, ciphertext, and key.

To measure AE quantitatively, the Avalanche Criterion (AC) is defined as:

(9) $$\mathrm{A}\mathrm{C}=\left(\frac{1}{n\times b}\sum _{i=1}^n\mathrm{H}\mathrm{D}(C_1^i,C_2^i)\right)\times 100\mathrm{\%}$$where $C_1^i$ and $C_2^i$ are ciphertext pairs generated by encrypting two keys differing by a single bit, $\mathrm{H}\mathrm{D}(\cdot )$ denotes their Hamming distance, $n$ is the total number of ciphertext pairs and $b$ is the length of the ciphertext in bits.

A robust encryption scheme should achieve an AC value close to 50%, indicating that on average half the ciphertext bits change due to a single bit key modification. Figure 4 illustrates this effect in the proposed scheme by demonstrating significant changes in ciphertext caused by minimal alterations in key.

Information entropy (IE)

IE measures the randomness or unpredictability of the intensities of pixels in an image. High entropy indicates that the image data exhibit no discernible patterns. On the other hand, the low entropy suggests that future pixel values can be predicted with greater certainty. IE is a key metric for evaluating the strength of an encryption method, as it helps to assess how well the encryption obscures the original image’s structure (Pizer et al., 1987; Tsai, Lee & Matsuyama, 2008; Gray, 2011; Volkenstein, 2009). The entropy value ranges from 0 to 8, with 8 representing a perfectly random image. In the case of an encrypted image, achieving an entropy value close to 8 indicates strong encryption and resistance to statistical attacks. The entropy of an encrypted image is calculated on the basis of the probability distribution of pixel intensities.

IE for an image can be calculated using the following equation:

(13) $$I_{\mathrm{E}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{y}}(\mathrm{I}\mathrm{m}\mathrm{g})=-\sum _{i=0}^{S-1}P({\mathrm{I}\mathrm{m}\mathrm{g}}_i){\log }_{2}P({\mathrm{I}\mathrm{m}\mathrm{g}}_i).$$

Here, $I_{\mathrm{E}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{y}}(\mathrm{I}\mathrm{m}\mathrm{g})$ represents the information entropy of the image $\mathrm{I}\mathrm{m}\mathrm{g}$, S is the total number of possible pixel values (or intensity levels), and $P(\mathrm{I}\mathrm{m}{\mathrm{g}}_i)$ denotes the probability that the pixel value $\mathrm{I}\mathrm{m}{\mathrm{g}}_i$ occurs in the image.

The results presented in Table 7 shows that the entropy values of the proposed encrypted images are close to 8, confirming their high degree of randomness and robustness against attacks.

Correlation coefficient (CC) analysis

The CC between adjacent pixels in an image indicates the degree of similarity in the intensities of the pixels. In the original image, the correlation between adjacent pixels is often high, as neighboring pixels typically share similar intensity values. To evaluate the effectiveness of an encryption method, we can analyze and compare the correlation coefficients of the original and encrypted (scrambled) images in various directions such as Horizontal (H), Vertical (V), and Diagonal (D) (Li, Du & Liang, 2020; Asuero, Sayago & González, 2006).

For a robust cryptosystem, it is essential that the correlation of an encrypted image in all directions is low, as this ensures that the encryption process has disrupted any predictable patterns. The mathematical computation for the CC between two adjacent pixel positions, which are denoted as R and S, is given by the formula:

(14) $${\mathrm{C}\mathrm{C}}_{R,S}=\frac{\mathrm{C}\mathrm{o}\mathrm{n}(R,S)}{\sqrt{\mathrm{D}\mathrm{e}\mathrm{v}}(R)\sqrt{\mathrm{D}\mathrm{e}\mathrm{v}}(S)}.$$

Here, R and S are two adjacent pixel positions, $\mathrm{C}\mathrm{o}\mathrm{n}(R,S)$ is the covariance between R and S, and $\mathrm{D}\mathrm{e}\mathrm{v}(R)$ and $\mathrm{D}\mathrm{e}\mathrm{v}(S)$ are the standard deviations of R and S, respectively, and W is the total number of pixel pairs analyzed.

The covariance and standard deviations are computed using the following equations:

(15) $$\mathrm{C}\mathrm{o}\mathrm{n}(R,S)=\frac{1}{W}\sum _{i=1}^W\left[(R_i-E(R))(S_i-E(S))\right]$$

(16) $$\mathrm{D}\mathrm{e}\mathrm{v}(R)=\frac{1}{W}\sum _{i=1}^W\left[{(R_i-E(R))}^2\right]$$

(17) $$\mathrm{D}\mathrm{e}\mathrm{v}(S)=\frac{1}{W}\sum _{i=1}^W\left[{(S_i-E(S))}^2\right]$$

(18) $$E(R)=\frac{1}{W}\sum _{i=1}^W{R}_i$$

(19) $$E(S)=\frac{1}{W}\sum _{i=1}^W{S}_i.$$

In these equations, $E(R)$ and $E(S)$ are the mean values of the pixel intensities for the sets R and S, respectively.

In an ideal cryptosystem, the correlation between adjacent pixels in the encrypted image should be minimal or close to zero, indicating that the encryption process has effectively randomized pixel relationships. This disruption of patterns makes it difficult for attackers to exploit the image’s structural information. A low CC indicates that the encrypted image has been successfully randomized, providing better resistance against statistical attacks that rely on the predictability of pixel distributions. In contrast, a high CC suggests that the encryption method may be weak, as the similarity between adjacent pixels could expose valuable information to an adversary.

Figure 5 shows the correlation of adjacent pixels in the H, V, and D directions for both the original and encrypted images. The numerical correlations are given in Table 8. The results clearly indicate that the CC of the encrypted image is significantly reduced compared to the original image, demonstrating the effectiveness of the encryption method in minimizing pixel predictability and improving security.

Histogram analysis

It evaluates the distribution of pixel intensities in an encrypted image. For an ideal robust cryptosystem, the histogram of the encrypted image should be uniformly distributed. It prevents any visible patterns or resemblances to the histogram of the original image, as illustrated in Fig. 6. This uniformity ensures that the encryption process effectively conceals the original data, making it highly resistant to statistical attacks (Zhang, 2021; Scott, 2010; Pizer et al., 1987).

Mathematically, the histogram of an image I can be expressed as

(20) $$H(I,v)={\sum }_{p,q}\delta (I(p,q)-v),v\in [0,255].$$

Here, $I(p,q)$ represents the pixel intensity at position $(p,q)$, $v$ is the pixel intensity level (ranging from 0 to 255 for an 8-bit grayscale image), $\delta (x)$ is the Dirac delta function, which is defined as:

(21) $$\delta (x)=\{\begin{array}{cc}1,\hfill & \mathrm{i}\mathrm{f}x=0\hfill \\ 0,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}.$$

Peak signal-to-noise ratio (PSNR) and mean squared error (MSE)

The performance of the decryption process in an encryption scheme is evaluated using two important image quality assessment metrics: MSE and PSNR. These metrics quantify the similarity between the decrypted image and the original image. A lower MSE value (close to 0) indicates that the decrypted image is very close to the original, with minimal distortion. A higher PSNR value (typically above 30 dB) suggests that the noise introduced during encryption and decryption is negligible and the visual quality of the image is preserved (Tanchenko, 2014; Joshi, Yadav & Allwadhi, 2016).

The mathematical formulation for MSE is given by the following:

(22) $$\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{L\times M}\sum _{p=0}^{L-1}\sum _{q=0}^{M-1}{\left(E(p,q)-O(p,q)\right)}^2.$$

Here, $E(p,q)$ and $O(p,q)$ represent the pixel values at position $(p,q)$ in the decrypted and original images, respectively. L and M denote the height and width of the image.

The corresponding PSNR is calculated using the following:

(23) $$\mathrm{P}\mathrm{S}\mathrm{N}\mathrm{R}=10\log \mathrm{\_}10\left(\frac{{255}^2}{\mathrm{M}\mathrm{S}\mathrm{E}}\right).$$

Table 9 illustrates PSNR values and MSE values close to their ideal values that confirm the effectiveness of the proposed encryption and decryption process. These results also validate that the decryption procedure can successfully reconstruct the original image with minimal loss or degradation.

Execution time

It is an essential factor to evaluate the feasibility of the model, particularly in scenarios where real-time encryption and decryption are required. A lower execution time indicates that the algorithm is efficient and can handle large image datasets with minimal delay. The execution time is influenced by several factors, including the complexity of the algorithm, the size of the input, and the available computational resources. To assess execution time, we measure the total time taken for both the encryption and decryption processes. The time is calculated in seconds and can be expressed as:

(24) $$T_{\mathrm{e}\mathrm{x}\mathrm{e}\mathrm{c}\mathrm{u}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}=T_{\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{r}\mathrm{y}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}+T_{\mathrm{d}\mathrm{e}\mathrm{c}\mathrm{r}\mathrm{y}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}.$$

Here, $T_{\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{r}\mathrm{y}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}$ and $T_{\mathrm{d}\mathrm{e}\mathrm{c}\mathrm{r}\mathrm{y}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}$ represent the time taken to encrypt and decrypt the image, respectively.

The encryption and decryption time of the proposed model are given in Table 9. It could be observed that the whole process is completed faster with a minimal computation time.

Table 10 provides a comprehensive comparison of the proposed image encryption algorithm against several recent methods using the standard Baboon image. The evaluation includes key statistical and cryptographic metrics such as the CC, IE, NPCR, and UACI. The CCs in H, V, and D directions for the proposed method are very close to zero (0.0025, −0.0030, and 0.0001 respectively) which indicates minimal similarity between adjacent pixels in the encrypted image. This reflects a strong level of diffusion, which is important to defend against statistical attacks. Similarly, other methods such as Rout & Mishra (2024) and Pati, Mishra & Rout (2024) exhibit relatively higher correlation values. This indicates weaker encryption effectiveness in breaking pixel relationships. The proposed method also achieves an IE of 7.9995 (close to the ideal value) that demonstrates high randomness and unpredictability in the encrypted image. It is slightly higher than the entropy values reported by other methods, which indicates superiority in resistance to entropy-based analysis. In addition, the proposed scheme shows excellent sensitivity to changes in the input image, as suggested by its NPCR and UACI values of 99.58% and 49.96%, respectively. Although NPCR is comparable to other methods, the UACI of the proposed approach is significantly higher. This indicates a much stronger response to minor changes in the original image and ensures better protection against differential cryptanalysis. Overall, the results clearly show that the proposed encryption technique delivers improved performance and security compared to other existing approaches.

The proposed framework strengthens the security posture of image-based systems by integrating multiple cryptographic layers. This strategy improves resistance to various cryptanalytic attacks, such as brute-force, differential, and statistical attacks. The simplicity of the approach makes it eligible for any domain without the need of significant modifications. The practical utilization of the approach is limitless but not confined to:

• Healthcare systems: for secure transmission and storage of sensitive medical images.

• Military and defense: where encrypted satellite or reconnaissance images must be protected under high-threat conditions.

• Cloud storage platforms: to ensure secure image backup and prevent unauthorized access or tampering.

• AI-based image analytics: to protect training datasets from reverse engineering or adversarial manipulation.

In addition, the modular, scalable, and adaptable design of the proposed encryption makes it suitable for integration into broader cybersecurity architectures. This study provides a reference framework for future research focused on securing multimedia data in evolving cryptographic threat landscapes.