Intelligent reflecting surface backscatter-enabled physical layer security enhancement via deep reinforcement learning

Related work

In Omid, Deng & Nallanathan (2022), the authors look into secure wireless communication in a system aided by an IRS, where one legitimate receiver’s communication is secured using the IRS in the presence of an eavesdropper. Next, they assume that the IRS is standalone, meaning that its passive beamforming operates entirely on its own. The authors in Yang et al. (2020) examine a wireless secure communication system that is assisted by an IRS. In this model, an IRS is utilized to adjust its reflecting elements in order to protect the communications of LU from being intercepted by multiple parties. In order to establish the best beamforming policy against eavesdroppers in dynamic environments, a unique deep post-decision state (PDS) prioritized experience replay (PER), PDS-PER, secure beamforming approach is proposed. This is because the system is very dynamic and complicated, making it difficult to handle the non-convex optimization problem.

This article (Lin et al., 2023) investigates the secure communication in a cooperative jammer-enabled Energy Harvesting based Cognitive Internet of Things (EH-CIoT) network. In the energy harvesting (EH) phase, the energy from the received radio frequency (RF) signals is first harvested by the jammer and the secondary transmitters (STs). The STs then transmit secret data to their intended recipients in the presence of eavesdroppers during the wireless information transfer (WIT) phase, during which the jammer sends a jamming signal to confuse the eavesdroppers. They derive the instantaneous secrecy rate and the closed-form expression of secrecy outage probability (SOP) to assess the secrecy performance of the system. Additionally, they provide a framework for joint energy harvesting time and transmission power allocation problems based on DRL. In article (Peng et al., 2022), authors examined a multi-user full duplex secure communication system along with IRS aided by hardware impairments at both the transceivers and IRS. An optimization problem is presented, that involves the transmit beamforming at the base station (BS) and the reflecting beamforming at the IRS. This problem aims to optimize the sum secrecy rate (SSR) while considering the transmit power restriction of the BS and the unit modulus constraint of the phase shifters. This computational problem is really difficult because the system has many dimensions and the environment keeps changing over time. The objective is to discover a suitable solution by utilising a DRL-based algorithm that gains knowledge from the ever-changing environment through repeated interactions.

Moreover, there are some closely related studies that have attempted to improve IRS based secure communication but couldn’t provide an outstanding convergence and data rate while maintaining the secure communication channel. The authors in Cao et al. (2023) proposed an approach to enhance backscatter in IRS for secure multiple input multiple output (MIMO) transmission, even in the presence of an eavesdropper and a suspicious jammer. They jointly design the IRS and active beamforming reflection coefficients of BS to maximize the system secrecy rate. Then nonconvex optimization problem has been solved by using an iterative block coordinate descent (BCD) technique. The majorization-minimization (MM) method was used to optimize the IRS backscatter coefficient matrix, while the Lagrange multiplier method has been used to optimize the active beamforming. This work (Liu et al., 2023) examines the PLS in an integrated sensing and communication (ISAC) system that assists multiple users and tracks a target concurrently, with the help of the IRS. In particular, they view the radar target as a possible eavesdropper who might obtain information about authorized users. Artificial noise (AN) is used to interfere with the reception of eavesdroppers. The authors collaborated in the design of the transmit beamforming, AN signal, and IRS phase-shift to optimize the achievable secrecy rate of all authorized users. They use the DRL algorithm to find the best learning strategy through agent and environment interactive learning because the problem is multi-variable coupling and non-convex optimization.

A novel hierarchical game framework with dynamic trilateral coalitions for PLS is examined in Chen et al. (2023). They investigated such complex and dynamic coalition relationships under the uncertainties of wireless systems (e.g., time-varying channel conditions) and formulated a hierarchical game integrated with a dynamic trilateral coalition formation game to model the strategic interactions among all three parties, i.e., LUs, Jammers and eaves in PLS. The authors specifically examine the trilateral coalition stability conditions first. They suggested an approach based on DRL for achieving equilibrium with long-term performance. In the context of future cyber-physical systems (CPS), where eavesdroppers may be present in the network and threaten the task offloading, this research (Chen et al., 2022) examines secure mobile edge computing (MEC). These eavesdroppers can function in two modes: one for collusion, in which they work together to decipher the secret communication, and another for non-collusion, in which each eavesdropper decodes the message independently. The authors in Wu & Zhang (2019) have provided a comprehensive study of DRL optimization of the IRS in MIMO communication systems. This article mainly focused in improving the spectral efficiency but it only considered the straightforward scenario. They have used the DRL-based deep deterministic policy gradient (DDPG) for the MIMO-based environment which has shown a better convergence and comparatively better spectral efficiency.

Although many modern research studies focus on the use of the IRS for security enhancement, they often overlook the complex relationship between eavesdropping and jamming attacks, and they also did not considered the IRS as a backscatter to improve the signal reception. The integration of IRS into wireless communication systems has emerged as a promising approach to enhance PLS. Recent advancements in IRS technology have demonstrated its potential to improve spectral efficiency, energy efficiency, and security in 5G-B5G networks (Mitev et al., 2023; Chen et al., 2019). IRS, with its ability to dynamically control electromagnetic waves, has been widely studied for its role in mitigating eavesdropping and jamming attacks (Cui, Zhang & Zhang, 2019; Shen et al., 2023).

Several state-of-the-art techniques have been proposed to leverage IRS for PLS. For instance, Omid, Deng & Nallanathan (2022) explored the use of IRS for secure communication in the presence of eavesdroppers, assuming standalone IRS operation. Similarly, Yang et al. (2020) proposed a DRL-based approach to optimize IRS beamforming for secure communication in multi-user scenarios. However, these works primarily focus on eavesdropping threats and do not account for the simultaneous presence of jamming attacks, which are common in real-world wireless networks. Despite these advancements, there are still significant gaps in the literature. Most existing works either focus on eavesdropping or jamming in isolation, and few studies have explored the joint optimization of IRS backscatter and active beamforming in the presence of both threats. Furthermore, the majority of current approaches assume perfect CSI, which is impractical in real-world scenarios due to channel estimation errors and user mobility.

Motivation and contribution

Since handling the eavesdropper and jammer simultaneously in terms of problem formulation is very challenging and complex, this article presents a novel method to improve IRS backscatter in a multiple-input single-output (MISO) communication system. This work (Yang et al., 2020) leverages IRS and DRL to achieve secrecy rates in the presence of eavesdroppers and multiple users but overlooks the adversarial impact of jammers, which can significantly compromise secrecy performance. However, while both eavesdroppers and jammers are considered in Cao et al. (2023), but the approach assumed perfect CSI and relies on the BCD-MM method, which is computationally complex and exhibits suboptimal convergence and security performance. While the proposed Deep-PLS aims to provide security against eavesdropping and jamming attacks by utilizing a DRL-based Deep-PLS framework to find the best convergence rate and the desired secrecy rate. The main contributions of this article are as follows:

• Our idea is the first of its type to strengthen the IRS backscatter to avoid malicious jamming attacks and eavesdropping attacks, simultaneously. This method reduces the influence of the unwanted jamming signal on LU and boosts the potential of the authorized channel through IRS backscatter and PLS, ultimately improving both security and data rate.

• We formulate the secrecy rate maximization problem by jointly optimizing the IRS backscatter coefficient matrix and the beamforming vector. This non-convex optimization problem of beamforming and Backscatter matrix is then solved using the DRL-based DDPG algorithm, aiming to enhance the secrecy rate.

• We proposed the DRL based Deep-PLS technique to enhance the security of the communication model. The central controller leverages real-time environmental observations and feedback to intelligently the beamforming policy using a DDPG algorithm, with secrecy rates as the reward function. Our proposed method outperforms existing techniques in terms of secrecy rate, demonstrating superior performance in improving security.

The other sections of this article are organized in the following way: “System Model” introduces the system model, discusses the approximation of channel estimation errors, and formulates the optimization problem. In “DRL-based Problem Mapping and Solution“, the DRL-based problem is formulated and the main components of the working algorithm are explained. “Simulation Setup” provides a detailed explanation of simulation environment and “Results and Discussion” justified the proposed scheme with detailed simulation results. Lastly, “Conclusions and Future Works” concludes the entire article.

Definitions and assumptions

Let $\mathrm{\Phi }=diag\left({\omega }_1{e}^{j\theta 1},{\omega }_2{e}^{j\theta 2},\dots ,{\omega }_l{e}^{j\theta l}\right)$ denotes the IRS reflection coefficient matrix, whereas ${\theta }_{\mathrm{l}}\in [0,2\pi ]$ and ${\omega }_{\mathrm{l}}\in [0,1]$ represent the phase shift coefficient and the amplitude reflection factor on the combined transmitted signal, respectively. Since it is intended for every phase shift to be planned to achieve complete reflection at the IRS. Specifically, ω and θ are used to control the signal reflection and modulation at the IRS. Thus, by carefully controlling $\mathrm{\Phi }$, we can use the jamming signal and modulate it to the LU to enhance the reception at the LU. It is to be noted that, unlike others, we are using the IRS to reflect the malicious jammer’s high frequency unwanted signal into the intended signal directed to the user, making this technique more implementable in terms of secrecy rate and achievable data rate requirements. The IRS controller has been designed to be intelligent, as we are applying a DRL-based algorithm to the controller. The IRS controller is behaving as a DDPG agent that is continuously interacting with the environment, and as a result of these actions, a reward is being added to the IRS controller, based on which the IRS controller will steer the signal’s reflection and modulation, improve the achievable SNR at the LU, and also maintain secrecy.

The signals received at the eavesdropper and LU respectively, are represented as follows:

(1) $$y_u={\mathbf{h}}_{bu}\mathit{w}s+{\mathbf{g}}_{ru}\mathrm{\Phi }\left({\mathbf{G}}_{br}\mathbf{w}+{\mathbf{H}}_{jr}q\right)s+{\mathbf{h}}_{ju}\mathbf{q}j+{\mathbf{n}}_u,$$

(2) $$y_e={\mathbf{h}}_{be}\mathit{w}s+{\mathbf{g}}_{ru}\mathrm{\Phi }\left({\mathbf{G}}_{br}\mathbf{w}+{\mathbf{H}}_{jr}q\right)s+{\mathbf{n}}_e.$$

The received noises at the LU and Eave are denoted by n_u and n_e, respectively. These noises at the user and eavesdropper are Gaussian distributed, respectively, with a zero mean, meaning that $n_u\sim \mathcal{C}\mathcal{N}\left(0,{\delta }_u^2\mathbf{I}\right)$ and $n_e\sim \mathcal{C}\mathcal{N}\left(0,{\delta }_e^2\mathbf{I}\right)$. Whereas $\mathcal{C}\mathcal{N}$ indicates the complex normal distribution of the random noise. In a complex normal distribution, the real and imaginary parts of the random variable are both normally distributed. Here, δ is a parameter representing the variance (or power) of the noise, and I is the identity matrix.

This modeling choice accurately reflects the nature of thermal and ambient noise in wireless communication systems, where both real and imaginary components are normally distributed. On the other hand, the exploration noise used in the DRL framework is generated using the Ornstein-Uhlenbeck (OU) process. Unlike Gaussian noise, OU noise is temporally correlated and designed to stabilize the learning process by preventing abrupt changes in action values. While the Gaussian noise pertains to the physical communication environment, the OU noise specifically addresses the training dynamics of the DRL algorithm, thereby serving two distinct yet complementary roles within our study. Alternative methods for modeling channel estimation errors, such as Autoregressive Moving Average (ARMA) processes or time-varying correlated noise, were initially disregarded due to their computational complexity and the challenges they pose for DRL training. These models require additional parameters to characterize temporal dependencies, which would increase the state space dimensionality and hinder convergence.

Keeping in view the received signal at the user and eavesdropper according to Eqs. (1) and (2), the following are the achievable data rates at both the user and the eavesdropper:

(3) $$R_{Lu}={\log }_{2}det(\mathit{I}+{\delta }_u^2{\mathbf{G}}_u{\mathbf{G}}_u^G\left(\mathit{I}+{\delta }_u^2{\mathbf{G}}_{ju}{\mathbf{Q}\mathbf{G}}_{ju}^G\right),$$

(4) $$R_e={\log }_{2}det\left(\mathit{I}+{\delta }_e^2{\mathbf{G}}_e{\mathbf{G}}_e^G\right).$$where, in the above equations, I is used for the identity matrix and δ² is used for the noise power of both the user and eavesdropper. In the channel matrices, we have ${\mathbf{G}}_u=$ ${\mathbf{h}}_{bu}\mathit{w}s+{\mathbf{g}}_{ru}\mathrm{\Phi }\left({\mathbf{G}}_{br}\mathbf{w}+{\mathbf{H}}_{jr}q\right)$ and ${\mathbf{G}}_e={\mathbf{h}}_{be}\mathit{w}s+{\mathbf{g}}_{ru}\mathrm{\Phi }({\mathbf{G}}_{br}\mathbf{w}+$ ${\mathbf{H}}_{jr}q)$. Now, we can calculate the secrecy rate of the user by simply taking the difference between the LU and the eavesdropper’s attainable data rate.

Mathematically expressed as follows:

(5) $${\mathbf{R}}_{\mathbf{s}}={\mathbf{R}}_{\mathbf{L}\mathbf{U}}-{\mathbf{R}}_{\mathbf{e}}$$

By substituting Eq. (3) and (4) into the (5), we get secrecy rate as:

(6) $$\begin{array}{c}{R}_s={\mathrm{l}\mathrm{o}\mathrm{g}}_2\mathrm{d}\mathrm{e}\mathrm{t}(\mathit{I}+{\delta }_u^2{\mathbf{G}}_u{\mathbf{G}}_u^G{\left(\mathit{I}+{\delta }_u^2{\mathbf{G}}_{ju}{\mathbf{Q}\mathbf{G}}_{ju}^G\right)}^{-1}-{\mathrm{l}\mathrm{o}\mathrm{g}}_2\mathrm{d}\mathrm{e}\mathrm{t}\left(\mathit{I}+{\delta }_e^2{\mathbf{G}}_e{\mathbf{G}}_e^G\right).\end{array}$$

The following are the main differences between IRS backscatter-aided anti-jamming, normal BCD-MM and our suggested Deep-PLS: When using a standard IRS-based PLS, the users’ and eavesdropper’s received signals are either strengthened or weakened by the IRS reflecting only the incoming signals, both wanted and undesirable. All incoming signals at the IRS are modulated into AN in IRS backscatter-aided anti-jamming, but this is done to protect against eavesdropping. Moreover, most of the work in IRS-absent PLS has been done using conventional numerical methods and other baseline optimization techniques. Significant progress has already been made in utilizing IRS in conjunction with PLS to enhance the security of wireless communication.

The authors in Cao et al. (2023) used the IRS-based PLS to secure wireless communication by solving the beamforming optimization problem at IRS and BS. But optimization problem was solved by the numerical technique BCD-MM. In this type of numerical technique, first we have to solve the convex optimization problem into some tractable form which is far more challenging to solve due to its complexity. In our work, we are using DRL-based methods to enhance the secrecy rate of the user. Also, unlike other baseline methods, we are using the IRS to use the high-frequency jamming signal to improve the SNR at the user by backscattering. Another challenge was to establish a steadfast high-speed link, e.g., a fiber link from BS to the IRS controller, to transmit the control signal. With this control signal, the IRS can control the reflection coefficient of each IRS element and steer the signal in real time. On contrary to it, we are applying the DRL-based algorithm to the IRS controller, making it intelligent itself, and thus eliminating the use of that dedicated fiber link. In this scenario, deciding whether the eavesdropper and jammer will behave in a colluding or non-colluding manner is important. If the jammer and eavesdropper are colluding, then the eavesdropper is aware of the jammer’s signal, and it can cancel the jammer’s signal. This poses a greater risk to LU because the eavesdropper and jammer are working collaboratively, ultimately reducing the SNR and secrecy rate of LU.

While in non-colluding behavior, the jammer and eavesdropper are unaware of their existence, and jammer signals also attack the eavesdropper, which deteriorates its ability to listen between BS and LU, thus enhancing secrecy rates. Considering the non-colluding behavior of the eavesdropper and jammer, this non-colluding assumption is grounded in realistic communication scenarios, where eavesdroppers and jammers typically act independently, often with different objectives and capabilities. The eavesdropper’s primary goal is to intercept confidential information without being detected, while the jammer’s objective is to disrupt the legitimate communication by introducing interference. By modeling them as non-colluding entities, we aim to reflect the challenges in designing robust communication systems that must protect against independent threats from both sides. This approach enhances the security analysis by considering the worst-case scenario, where the adversaries do not cooperate, thereby providing a more generalized and practical solution that aligns with real-world adversarial conditions.

Channel estimation errors and its approximations

The BS and IRS may not be able to acquire the perfect CSI in practical systems. This is because both the transmission and processing delays are present, together with the user’s mobility (Wu & Zhang, 2019). Therefore, the CSI is considered to be imperfect when the BS and the IRS transmit the data stream to the user equipment. This outdated CSI for beamforming could have an adverse effect on the demodulation process for the mobile units (MUs), ultimately resulting in significant performance degradation (Hong et al., 2023). Additionally, although imperfect CSI is used to demonstrate the core functionality of the Deep-PLS method, it may not fully capture the complexities present in real-world environments. Imperfect CSI, which arises due to factors such as quantization errors, estimation delays, and noise, can significantly impact the performance of the proposed method. Thus, considering these real-world things and imperfect CSI, setting the suitable value of noise makes it more attractive to be implemented in real world environment. Hence, it is imperative to take into account the obsolete CSI in the IRS-assisted secure communication system. Therefore, the CSI of actual channels G_br, h_bu, h_bp, g_ru, g_rp, H_jr, h_ju, h_jp is represented as:

(7) $$\begin{array}{rl}& {\mathbf{h}}_{bu}={\tilde{\mathbf{h}}}_{bu}+{\overline{\mathbf{h}}}_{bu},\\ & {\mathbf{h}}_{bp}={\tilde{\mathbf{h}}}_{bp}+{\overline{\mathbf{h}}}_{bp},\\ & {\mathbf{g}}_{ru}={\tilde{\mathbf{g}}}_{ru}+{\overline{\mathbf{g}}}_{ru},\\ & {\mathbf{g}}_{rp}={\tilde{\mathbf{g}}}_{rp}+{\overline{\mathbf{g}}}_{rp},\\ & {\mathbf{h}}_{jr}={\tilde{\mathbf{h}}}_{jr}+{\overline{\mathbf{h}}}_{jr},\\ & {\mathbf{h}}_{ju}={\tilde{\mathbf{h}}}_{ju}+{\overline{\mathbf{h}}}_{ju},\\ & {\mathbf{h}}_{jp}={\tilde{\mathbf{h}}}_{jp}+{\overline{\mathbf{h}}}_{jp}.\end{array}$$where ${\tilde{\mathbf{h}}}_{bu}$ are the estimated channel’s errors and ${\overline{\mathbf{h}}}_{bu}$ are channels estimation ratio matrices, respectively. Also, it will be the same for everyone else in channel coefficients. We will consider the channel errors as white Gaussian noises and ψ is applied to show the ratio of channel error estimation. Consequently, every component of ${\tilde{\mathbf{h}}}_{bu},{\tilde{\mathbf{h}}}_{bp},{\tilde{\mathbf{g}}}_{ru},{\tilde{\mathbf{g}}}_{rp},{\tilde{\mathbf{h}}}_{jr},{\tilde{\mathbf{h}}}_{ju}$ and ${\tilde{\mathbf{h}}}_{jp}$ will be $\mathcal{C}\mathcal{N}\left(0,{\psi }^2{\beta }_{H_{bu}}\right),$ $\mathcal{C}\mathcal{N}\left(0,{\psi }^2{\beta }_{H_{bp}}\right),\mathcal{C}\mathcal{N}\left(0,{\psi }^2{\beta }_{g_{ru}}\right)$ and $\mathcal{C}\mathcal{N}\left(0,{\psi }^2{\beta }_{g_{rp}}\right),$ $\mathcal{C}\mathcal{N}\left(0,{\psi }^2{\beta }_{H_{jr}}\right),\mathcal{C}\mathcal{N}\left(0,{\psi }^2{\beta }_{H_{ju}}\right),\mathcal{C}\mathcal{N}\left(0,{\psi }^2{\beta }_{H_{jp}}\right)$ respectively, where ${\beta }_{H_{bp}}$ represents the path loss between the BS and eavesdropper, and the same goes for all.

Since the channel estimation errors are assumed to be white Gaussian, the secrecy rate will now be:

(8) $$R_s=\mathrm{l}\mathrm{o}{\mathrm{g}}_2\mathrm{d}\mathrm{e}\mathrm{t}\left[\left(\mathbf{I}+{\mathbf{G}}_u{\mathbf{G}}_u^G{\left({\mathrm{\Omega }}_u+{\mathbf{G}}_{ju}{\mathbf{Q}\mathbf{G}}_{ju}^G\right)}^{-1}\right)-\mathrm{l}\mathrm{o}{\mathrm{g}}_2\mathrm{d}\mathrm{e}\mathrm{t}\left(\mathbf{I}+{\mathbf{G}}_e{\mathbf{G}}_e^G{\left({\mathrm{\Omega }}_e\right)}^{-1}\right)\right].$$

${\mathrm{\Omega }}_{\mathrm{u}}$ and ${\mathrm{\Omega }}_{\mathrm{e}}$ represent the covariance matrices for the related noise for all channels. Because we have assumed the white Gaussian noises, so the diagonal elements of g_ru diag ( $\tilde{\mathbf{G}}br$) w and covariance matrices and other elements approximated as zero; see detailed derivation in Cao et al. (2023). To capture a more realistic channel model, we extend the traditional Gaussian error assumption by incorporating temporally correlated channel estimation errors. Specifically, the imperfect CSI for the channel between the BS and the LU is modeled as:

(9) $${\hat{\mathbf{h}}}_k\left(t\right)={\mathbf{h}}_k\left(t\right)+{\mathrm{ϵ}}_k\left(t\right),$$where ${\mathrm{ϵ}}_k\left(t\right)$ is modeled as a first-order autoregressive $\left(\mathrm{A}\mathrm{R}\left(1\right)\right)$ process:

(10) $${\mathrm{ϵ}}_k\left(t\right)=\rho {\mathrm{ϵ}}_k\left(t-1\right)+\sqrt{1-{\rho }^2}{\eta }_k\left(t\right).$$with ${\mathit{\eta }}_k\left(t\right)\sim \mathcal{C}\mathcal{N}\left(0,{\sigma }_{\mathrm{ϵ}}^2\mathbf{I}\right)\mathrm{a}\mathrm{n}\mathrm{d}\rho \in \left[0,1\right]$ representing the temporal correlation coefficient due to Doppler effects. When ρ = 0 this model reduces to the conventional i.i.d. Gaussian error assumption. Therefore, our suggested Deep-PLS-based method can obtain near-optimal outcomes and become a practical technique, assuming the imperfect CSI as the white Gaussian noise. With these channel error approximations and assumptions, the proposed Deep-PLS-based agent can achieve the desired secrecy rate.

Problem formulation

Our primary goal is to optimize the secrecy rate while maintaining a good level of SNR at LU by backscattering and PLS. This can be achieved by optimizing the active beamforming ω at BS and the backscattering coefficient matrix φ at IRS. The IRS, in this case, is also responsible for backscattering the high-frequency signal of the jammer to improve the reception for the user while maintaining security of the data stream R_s at the LU. The goal is to establish secure wireless communication between BS and LU and improve reception at LU by backscattering by the IRS. Therefore, from a computational standpoint, the problem can be expressed as:

(11a) $$PO\underset{w,\mathrm{\Phi }}{\mathrm{m}\mathrm{a}\mathrm{x}}C\left(w,\theta ,\mathrm{\Phi },H\right)$$ (11b) $$\mathrm{s}.\mathrm{t}.w⩽P_t$$ (11c) $$\left|{\omega }_l{e}^{j\theta l}\right|=1,0<{\theta }_l⩽2\pi ,1⩽l⩽L$$where P_t is the transmit power of the BS. In Eq. (11a), our objective function is described, which is to optimize the backscatter matrix, amplitude, and phase of the IRS to maximize the secrecy rate by beamforming at the IRS. In Eq. (11b), the limitation in the maximum power constraint of the BS is satisfied. Similarly, Eq. (11c) describes phase and amplitude ranges. As the objective function in Eq. (11a) is non-convex concerning either w or ɸ, and the coupling of both optimization variables w and ɸ and, it is evident that finding an optimal solution to optimization Eq. (11b) is difficult. we considered a robust beamforming architecture designed to maximize the system’s minimum achievable secrecy rate, while ensuring that all worst-case constraints are satisfied.

The main components of proposed algorithm

• •

  Critic network: The critic network as shown in Fig. 2 learns the Q-function $\mathrm{Q}(s,a|{\theta }^Q)$, which estimates the expected return when taking action, a from state s and then following the policy µ. The parameters of the critic network are denoted by ${\theta }^{\mathrm{Q}}$. (15) $$Q(s,a|{\theta }^Q)=\mathbb{E}[r+\gamma Q(s^{\prime }),\mu (s^{\prime }|{\theta }^{\mu })|{\theta }^Q].$$

• •

  Replay buffer: The replay buffer D stores past experiences (s, a, r, ś) of the agent. During training, experiences are sampled uniformly from the replay buffer to break the correlation between experiences and stabilize learning.

• •

  Target networks: DDPG uses target some networks to stabilize learning process. These networks, denoted ${\mu }^{\prime }$ and $Q^{\prime }$ are the copies of the actor and critic networks, respectively, with slowly updated parameters. (16) $$\begin{array}{r}{\theta }^{\prime \mu }\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{\prime \mu },\\ {\theta }^{\prime Q}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{\prime Q}.\end{array}$$

• where $\tau$ is the soft update rate.

• •

  Noise exploration: Noise exploration encourages exploration by adding noise to the actions chosen by the actor. The noise is typically sampled from a stochastic process such as the Ornstein-Uhlenbeck process. The last part of this equation shows the noise sampled. (17) $$a^{\prime }=\mu (s|{\theta }^{\mu })+\mathcal{N}.$$

• •

  Update rules: Both the actor and critic networks are updated using Stochastic Gradient Descent (SGD) based on sampled experiences from the replay buffer.

• •

  Actor update: The actor aims to maximize the expected return by adjusting the policy parameters θ µ. The gradient of the actor’s objective function is estimated using the sampled experiences.

  (18) $$\begin{gathered} {\mathrm{\nabla }}_{\theta \mu }J\approx \frac{1}{N}\sum _i{\mathrm{\nabla }}_{a}Q(s,a|{\theta }^Q){|}_{s=s_i,a=\mu (s_i)}{\mathrm{\nabla }}_{\theta \mu }\mu (s|{\theta }^{\mu }){|}_{s_i} \\ {\theta }^{\mu }\leftarrow {\theta }^{\mu }+\alpha {\mathrm{\nabla }}_{\theta \mu }J. \end{gathered}$$

  where J is the expected return, N is the batch size, and α denotes the learning rate of the actor.

• •

  Critic update: The critic is updated by minimizing the Temporal Difference (TD) error between the predicted Q-value and the target Q-value. (19) $$\begin{array}{rl}\mathrm{T}\mathrm{D}\mathrm{E}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}=& Q(s,a|{\theta }^Q)-(r+\gamma Q(s^{\prime },\mu (s^{\prime }|{\theta }^{\mu })|{\theta }^{Q\prime }))\\ & \mathcal{L}({\theta }^Q)=\frac{1}{N}\sum _i(\mathrm{T}\mathrm{D}\mathrm{E}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}{)}^2.\\ & {\theta }^Q\leftarrow {\theta }^Q-\beta {\mathrm{\nabla }}_{{\theta }^Q}\mathcal{L}\end{array}$$

• where $L({\mathrm{\theta }}^{\mathrm{Q}})$ is the loss function for the critic, and β is the learning rate for the critic.

Secure beamforming training with DDPG

For the BS, the central controller is a learning algorithm that initially requires training to perform the essential tasks related to environmental data collection and decision-making. During the training stage, the controller initializes network parameters and collects crucial information about the current system state as shown in Algorithm 1. Typically, this data integrates the CSI of LU, the previously predicted secrecy rate, and the transmission data rate. Once the state vector, composed of this gathered information, is prepared, it is fed into a DDPG algorithm for training the learning model. DDPG, being an actor-critic model, employs both an actor network to decide on actions and a critic network to evaluate them. The algorithm iteratively updates the parameters by interacting with the environment and refining its policy to maximize rewards.

Unlike Q-learning, which uses discrete action spaces, DDPG is well-suited for continuous action spaces, making it ideal for secure beamforming tasks where actions, such as adjusting transmission power or beamforming vectors, are continuous. The agent performs actions based on the current state, and the critic evaluates the action’s effectiveness by estimating the value function. The agent’s policy is continuously refined by minimizing the temporal difference error. In this context, the exploration-exploitation trade-off is managed by introducing noise to the action selection process (OU noise), allowing the agent to explore new areas of the action space and improve its policy. After selecting an action, the agent interacts with the environment, receives a reward, and transitions to the next state s_t+1. This iterative process enables the agent to improve its decision-making ability for secure beamforming by analyzing the rewards and system states it encounters. In summary, training the central controller using DDPG allows it to effectively learn how to optimize secure beamforming in wireless communication systems. This is achieved by learning a continuous policy that balances exploration of new strategies with the exploitation of the best-known actions to maximize long-term rewards.

Computational complexity analysis

To highlight the practical advantages of the proposed DRL-based scheme over traditional iterative optimization techniques such as BCD-MM, we perform a brief complexity analysis. BCD-MM typically requires solving convex sub-problems in each iteration, and its overall complexity is in the order of $\chi$(I.L³), where I is the number of iterations and L is the number of IRS elements or optimization variables. Moreover, the convergence speed of BCD-MM can vary significantly based on the initial conditions and system parameters. This iterative nature, combined with sensitivity to initialization and real-time adaptability challenges, limits its practicality in highly dynamic environments such as IRS-assisted 6G networks. In contrast, the DRL-based approach incurs a one-time training cost, after which real-time decision-making is achieved with negligible computational delay. The online complexity of DRL (e.g., DDPG) is approximately O(E. T(ɣ. N)) where E is the number of Episodes, T is the time steps and ɣ is the number of layers in the neural network and N is the number of neurons per layer. Once trained, the policy network can infer optimal actions in real-time, making it highly scalable and suitable for dynamic and time-sensitive environments such as THz communications with mobile users. Furthermore, our empirical results (e.g., Figs. 3–6) demonstrate superior convergence, secrecy rates, and adaptability of Deep-PLS even with imperfect CSI and in the presence of jamming. These advantages make the proposed DRL method a practical and scalable solution for secure IRS-assisted communications in real-time 6G scenarios.

Hence, although DRL may initially require higher offline training effort, its online inference capability and adaptability offer significant benefits over iterative solvers, particularly in scenarios demanding real-time responsiveness and reconfigurability. For example, compared to BCD-MM’s average secrecy rate of 2.8 bps/Hz, our Deep-PLS achieves 3.57 bps/Hz under identical conditions, confirming the superiority not only in performance but also in scalability and computational feasibility.

Simulation setup and parameters

The BS is equipped with 4 antennas, represented by the variable M = 4. These antennas are used to transmit signals to LU and eavesdroppers. This setup impacts the beamforming and transmission power. The IRS is a critical component in enhancing the signal quality and reducing interference. It is composed of 60 elements (E = 60), at first then we take results iterating the IRS elements from 40 to 100 IRS elements as shown in fig. The IRS elements help manipulate the signal path to improve the reception at the legitimate users while minimizing eavesdropping. The BS operates at a relatively low transmit power. This value is intended to simulate realistic power levels for wireless communication systems while taking path loss and interference into account. The exponent value for path loss is set as ρ = 2, which is typical for urban environments. The reference distance for path loss calculations is set to 1 m (d₀ = 1). The path loss function computes the loss of signal power over distance. This function is used throughout the simulation to model the channel’s impact on the received signal strength at various distances.

Actor-critic neural networks

The actor network is responsible for selecting actions (IRS phase shifts) based on the current state of the system. The network takes in the state vector, which includes both the number of IRS elements and antennas at the BS. This is the input to the actor network. The proposed method leverages DDPG approach to optimize secure beamforming in IRS-assisted communication systems. The neural network architecture consists of an actor network with three hidden layers (512, 256, and 128 neurons) and a critic network with two hidden layers (256 and 128 neurons), both employing ReLU activations and trained using the Adam optimizer with a learning rate of 1e⁻³. The DDPG algorithm is configured with a discount factor ( $\mathit{\gamma }$) of 0.95, batch size of 64, replay buffer size of 10,000, and target network update rate ( $\mathit{\tau }$) of 0.005. An initial exploration noise of 0.5 is used, decaying over time to 0.999 to balance exploration and exploitation during training. The simulation environment is designed to generate dynamic channel conditions, including path loss and noise variance, reflecting real-world scenarios with imperfect CSI. The DRL agent adapts to the time-correlated CSI noise by updating its policy using temporally smoothed observations, which better reflects real-world conditions such as mobility and Doppler shifts.

The reward function is carefully designed to maximize the SNR difference between legitimate users and eavesdroppers while minimizing the potential for eavesdropping. The training process comprises 1,500 episodes, each with 300 time steps, where the model learns optimal beamforming strategies through continuous interaction with the environment.

The simulation experiments were conducted on a Dell i7-11th generation system running Windows 11 using the Anaconda Python environment (version 3.11.3). The deep reinforcement learning model was implemented using the PyTorch library (version 2.0.1), which provided efficient handling of neural network training and backpropagation. Additionally, numerical computations and data handling were managed using NumPy (version 1.24.3), while data visualization and plotting were performed using Matplotlib (version 3.7.1). The simulation code leverages the random library for generating stochastic noise and initial conditions. Furthermore, the torch.optim module was employed to optimize the actor and critic networks using the Adam optimizer. To ensure the stability of the training process, the torch.nn module was used to construct the neural network architecture with ReLU activation functions.

Initially, high exploration noise is added to the actions taken by the actor to ensure sufficient exploration of the action space. Over time, this noise decays to encourage the agent to exploit learned behaviors. The reward is based on the SNR difference between the LU and the eavesdropper’s signal.