Enhanced recurrent attention-deep Q learning with optimal node constrains and effective penalty based model for data transmission scheduling on wireless sensor networks

Role of reinforcement learning

RL plays a critical role in enhancing real-time data transmission scheduling in WSNs. RL is a machine learning paradigm in which an agent learns to make decisions by interacting with its environment and receiving feedback in the form of rewards or penalties. In the context of WSNs, RL can dynamically adjust transmission schedules in response to changing network conditions, thereby improving efficiency and reliability. The RL framework comprise an agent, a state space, an action space, a policy, and a reward function. The agent makes decisions (actions) to maximize accumulated rewards over time, while the policy defines the strategy for selecting actions in various states. The reward function evaluates the desirability of the agent’s behaviour. (1)${\displaystyle Q\left(s,a\right)=\left(1-\propto \right).Q\left(s,a\right)+\propto \cdot \left[r+\gamma \cdot {max}_{a}Q\left(s^{{}^{\prime }},a\right)\right]}$

Here, $Q\left(s,a\right)$ quantifies the appropriateness of actions a in state s, r is the immediate reward, ∝ is the learning rate, γ isthe discount factor, and s′ is the next state.

Wireless sensor networks in data transmission

WSNs serve as a backbone for real-time data transmission, enabling seamless communication sensor nodes. Efficient data transfer in WSNs is crucial for applications such as environmental monitoring, healthcare, and smart cities. Medium access control (MAC) and time division multiple access (TDMA) are two fundamental components in the management of wireless network data transmission.

Deep Q-learning fundamentals

DQL stands at the forefront of reinforcement learning methodologies, transforming the optimization of decision-making processes in dynamic environments. Fundamentally, DQL integrates the capabilities of deep neural networks and Q-Learning, enabling agents to learn optimum strategies through interaction with their surroundings. In the context of real-time data transmission scheduling, DQL offers a robust framework for dynamically allocating resources in WSNs, ensuring timely and efficient communication between sensor nodes.

In the DQN network framework, the neural network functions as a supervised learning mechanism for WSNs. The general approach constructs two Q networks, where the target Q value defines the loss function. The input pool provides training samples, and the parameters W and the bias b are updated using the stochastic gradient descent method (SGD), which also calculates the gradient after estimating Q value. This process illustrates the structure of the DQN network model.

Figure 1 illustrates two networks constructed with the same topology but exhibiting different characteristics. One network predicts and calculates the Q value is using the most recent parameters, while the other updates the Q value based on parameters from an earlier point in time. This approach enhances the algorithm’s reliability more reliable, ensures greater stability of the target Q-value over a period, reduces the correlation between the current and target Q values. The experience pool, also known as experience replay, serves not only to provide training examples but also to address the issue of data correlation. At the outset of the training process, a memory bank is initialized. The experience pool stores the current state, action, reward, and the resulting state in the next time slot following the execution of an action. During network training, a specified amount of memory is utilized, with mini-batches randomly sampled from the experience pool. Furthermore, once the experience pool reaches capacity, new experiences overwrite the oldest ones, disrupting the original data sequence and further minimizing information correlation.

Networking structure

WSN features a predefined topology and consists of M sensor nodes and one base station node. The sensor nodes are responsible for data generation and transmission tasks, while the base station functions as the destination node. Sensor nodes periodically generate data with specific deadlines and transmit it to the base station through inter-node communication within designated time slots.

Packet representation of data

The data packet generated by the i-th node, denoted as P_i, is defined as: P_i = T_i, H_i, D_i, ϕ_i. In this case, T i represents the packet generation interval, D_i is the packet transmission deadlines ϕ_i indicates the routing path of the packet, and H_i denotes the total number of hops from the source node to the sink node. The parameters T_i and D_i are expressed in units of time slots. Typically, to ensure sufficient time for successful transmission T_i must be greater than H_i.

Packet properties at time slot

At any given time slot t, the data packet is characterized by three key attributes: C_i, h_i, t_i. (C_iϵϕ_i), represents the current node location of the data packet and h_i.indicates the number of remaining hops in the transmission process. (0 < t_i. ≤ T_i.) denotes the remaining time to meet the transmission deadline.

Method of transmission

Transmission is considered feasible if t_i. > h_i.. When a time slot is reserved for transmission, both the remaining time and the number of hops are reduced by one, facilitating progress to the next node. If transmission does not occur, only the remaining time t_i. is reduced by one. When t_i = 0, the node generates a new data packet and enters a waiting phase for transmission scheduling, initializing with t_i = T_i. and h_i = H_i.

The proposed model is based on the following realistic assumption to guide the practical design and implementation of WSNs:

1. Exclusive transmission capability: sensor nodes are not capable of simultaneous data transmission and reception, nor can they receive from multiple nodes concurrently.

2. Single data packet selection: to ensure effective resource use, nodes receiving multiple data packets can only choose one packet to transmit during each time period.

3. Strict cut-off time for data: in order to preserve synchronization, data created by source nodes on a regular basis are subject to a stringent cut-off time constraint that corresponds with the data creation cycle.

4. Immediate discard for exceeded deadlines: to avoid transmitting out-of-date or unnecessary information, a data packet is immediately deleted if its transmission deadline is exceeded.

5. Effect of physical elements on communication: transmission power, encoding style, and modulation scheme are examples of physical elements that affect the likelihood of a successful wireless communication transmission.

6. Probability of performance in transmission: the modeling method is made simpler while admitting the unpredictability in wireless communication performance by assuming that if data is organized for transmission, the likelihood of success is 1.

7. Time constraints are taken into account: it is acknowledged that data transmission is time-sensitive, guaranteeing prompt delivery of information within designated deadlines.

8. Optimization for resource efficiency: by maximizing resource allocation and usage within the bounds of realistic WSN installations, the assumptions seek to enhance network performance and data transmission efficiency.

State representation

The state in your system should include information that the agent (the deep RNN) can use to make scheduling decisions. A suitable state representation in the existing model for data transmission scheduling in WSNs

1. Buffer status: information about the data buffer status of each sensor node. For example, the number of packets waiting to be transmitted.

2. Network conditions: data related to the current network conditions, such as the channel occupancy, signal strength, interference, and available bandwidth.

3. Historical schedules: information about past scheduling decisions and their outcomes, including success rates and energy consumption.

4. Data urgency: priority of data transmission

This state information at time step t can be represented as S(t). In this work different kind information about each sensor node is represented by the state. Those are described as follows.

A. Number of hops

A fundamental characteristic of a node in a WSN is the number of hops, or intermediate relay nodes, required to transmit data along the communication path between the originating node and the target node. Due to the limited signal amplification capabilities inherent in sensor nodes, direct communication is typically impractical. Consequently, multi-hop communication becomes indispensable for enabling long-range data transmission within WSNs. Depending on the network design, data can be transmitted using either a few long distance hops or several short-distance hops. Each transmission strategy long hop or short hop offers distinct advantages. Long-hop transmission minimizes the number of participating nodes, thereby reducing routing complexity and associated overhead costs. Conversely, short-hop transmission is more energy-efficient, as the energy required for wireless transmission increases with distance. Therefore, selecting the appropriate hop strategy involves balancing trade-offs between transmission cost and energy consumption.

B. Packet life time

All data packets at nodes should have limited lifetime. The lifetime represents its actual time period from source to destination.

C. Node degree

Node degree is common parameter used in all WSN scheduling as well as routing. It estimated for each node based on the number of one-hop neighbours connected to it. If a node has a higher node degree, it can transmit the data to more neighbours, which likely increases the success rate of transmission. Node degree is represented as N_deg and can be expressed follows: (4)${\displaystyle N_{\text{deg}}\left(N_i\right)=\parallel N_j\parallel wherej\in \text{one hop neighbour of}{N}_i}$

D. Node energy and average energy of neighbour nodes

Due to limited energy resources and distances between nodes, communication between sensor nodes and sink occurs via a multi-hop model. Each node forwards the collected measurements to neighbouring nodes, which subsequently relay them further. Monitoring the energy consumption of active sensor nodes allows applications and routing protocols to make intelligent, data-driven decisions, thereby enhancing the overall operational longevity of the sensor network.

In this module, the current node’s energy and the average energy derived from its neighbouring nodes are maintained as state parameters. Information such as residual energy, node location, and the type and size of packets from one-hop neighbouring nodes is used to calculate the average energy of neighbouring nodes. Each node records the information of its neighbour upon receiving a packet. Once neighbouring nodes are identified, the node calculates their average energy as follows (5)${\displaystyle E_{avg}\left(N_i\right)=\frac{\sum _{j\in Neg\left(N_i\right)}Energy\left(N_j\right)}{\parallel Neg\left(N_i\right)\parallel }}$ where Eavg(i) is the average energy of neighbouring node i, $Neg\left(N_i\right)$ is the set of neighbour nodes, $\parallel Neg\left(N_i\right)\parallel$ is the number of neighbors, j is the neighbour node of node i in Eq. (5)

E. Node local balancing and spatial coverage parameters

In WSN, each node may have a varying number of neighbours and may be distributed in different spatial patterns. Therefore, determining balanced neighbour coverage is a key factor in ensuring efficient data transmission. This metric indicated whether nodes are uniformly distributed near the source node or whether there is significant variation in their spatial arrangement. From this value, it is possible to determine if certain nodes are closer to the boundary of the coverage area, an aspect that enhances the likelihood of identifying the next hop node and helps avoid communication voids. This module extracts two values to characterize the distribution of the neighbouring nodes as local balancing coverage (LBC) local spatial coverage (LSC).

As illustrated in Fig. 3, the neighbour nodes of n1 and n2 fall within the transmission range r. In terms of local balancing coverage, node n1 exhibits better distribution than node n2, as its neighbours are positioned at relatively equal distances. In contrast, node n2 has neighbours at varying distances, indicating an uneven distribution. The local balancing coverage for a node ‘i’ is calculated as follows: firstly, the distance between each neighbour node ‘j’ from node ‘i’ is computed and stored in a list called DList. Then, for each entry in DList, the average distance deviation (ADist_j) is calculated and compiled into a list DList as ADList. Finally, the LBC is derived as the ratio of the difference between the maximum and minimum values in ADList to the transmission range r. (6)${\displaystyle {Dist}_j=\sqrt{{\left(x_i-x_j\right)}^2+{\left(y_i-y_j\right)}^2},wherej\leftarrow Neg\left(N_i\right)}$ (7)${\displaystyle {ADist}_j=abs\left({Dist}_j-Mean\left(DList\right)\right)}$ (8)${\displaystyle LBC\left(N_i\right)=\frac{\left(max\left(ADList\right)-min\left(ADList\right)\right)}{r}.}$

Let us consider the node n1 has five neighbour nodes in the distance of 105, 100, 110, 115 and 90 from the position of n1 within the transmission range 200 and its average is 104. Then for each neighbour the ADist_j is estimated using the Eq. (7) and from that the $LBC\left(n1\right)$ is estimated using the Eq. (8) which is 29.9. This value indicates that the nodes are covered in balanced range from n1 compared to the $LBC\left(n2\right)$ which is around 50, where the neighbour nodes are in the distance 190, 180, 90, 190 and 60 from n2.

Similarly the local spatial coverage is estimated using the following Eq. (9). The detected $LSC\left(n_1\right)$ is 0.52 which shows the nodes are more or less in half radius from its coverage, whereas the $LSC\left(n_{21}\right)$ is 0.71 it reflects most of the nodes are nearer to boundary region. It is clearly noted in the Fig. 3. (9)${\displaystyle LSC\left(N_i\right)=Mean\left(DList\right)/r.}$

F. Joined link coverage score

In this study, an additional metric joined link coverage score is introduced. This score evaluates the cumulative interconnectivity of each neighbouring node with other neighbouring nodes, based on the number of shared one-hop neighbours of node i. Accordingly, it is termed joined link coverage. The computation begins by determining the reciprocal of the number of common neighbours of each one hop neighbour and the others. The average of these values is then calculated to yield the JLCS. The estimation is presented in Eq. (10). Nodes with more interconnected neighbours receive a lower score compared to those whose neighbours are parsley connected. This metric is instrumental the scheduling overhead in the network. (10)${\displaystyle JLCS\left(N_i\right)=mean\left(\frac{1}{\parallel Neg\left(N_j\cap Neg\left(N_k\right)\right)\parallel }\right)}$ where j, k ← Neg(Ni)

G. Hop waited score

Normally the packet adjacency score of a node where at presents is estimated with the help of the number of hops to travel and the time to live or lifetime of the packet which is estimated as follows: (11)${\displaystyle US\left(N_i\right)=\left(\frac{\text{Number of Hops to Travel}}{\text{Life Time of the Packet}}\right).}$ This urgency score serves as one of the input parameters for the DQN model. Once calculated for the current state, the algorithm compares all the urgency scores of all nodes. In traditional models, the node with the highest urgency score typically receives the highest scheduling priority. However, this algorithm introduces a more nuanced approach by evaluating whether other nodes are at risk of missing a valuable scheduling opportunity. The procedure is described as follows: $US\left(N_k\right)$ represent the highest urgency score among all ‘n’ nodes in the network. The algorithm compares this value with the urgency scores of the remaining nodes. If any node’s urgency score falls within a specific threshold of the maximum urgency score, that node is also considered for scheduling. Subsequently, the hop-ratio of the considered node is calculated relative to the hop count of node ‘k’. If this ratio is less than 60%, the urgency score is further increased by applying a hop-based weighting factor. ${\displaystyle If\left(US\left(N_i\right)-US\left(N_m\right)\right)\le 0.2\text{then}:}$ ${\displaystyle HDev=\left(\frac{\text{Number of Hops to Travel from}{N}_m}{\text{Number of Hops to Travel from}{N}_i}\right)}$ ${\displaystyle IfHDev\le 0.6then:}$ node m′ selected for urgency score adjustment (12)${\displaystyle US\left(N_m\right)=US\left(N_m\right)+\left(US\left(N_i\right)\ast \left(1-HDev\right)\right).}$ From this itself, the Hop weighted Missing Score (HWMS) is also estimated, which is used for the penalty estimation of DQN. The hop weighted missing score is estimated after the action state as, if the current state is not allotted even though it attains hop based weight score means 30% of its urgency score is assigned as the hop weighted missing score.

In this work reward and penalty is estimated with the constraints values and scores. (13)${\displaystyle \mathrm{Reward}=w1\ast Reward=w1\times US+w2\times \frac{1}{\mathrm{Eavg}}+w3\times \frac{1}{\mathrm{LBC}}}$ where w1 + w2 + w3 = 1 (14)${\displaystyle \mathrm{Penalty}=w1\times \frac{1}{JLCS}+w2\times \frac{1}{\mathrm{HWMS}}.}$

WSN-DQL-BiLSTM attention-based architecture for data transmission scheduling

This section presents the wireless sensor network-deep Q learning bidirectional long short-term memory (WSN-DQL-BiLSTM) WSN-DQL-BiLSTM attention-based architecture, developed for real-time data transmission in WSNs, context-driven attention layers combined with bidirectional sequence, the architecture is designed to effectively capture temporal dependencies and emphasize relevant features within the incoming data. The model enhances the network’s ability to make informed scheduling decisions by combining attention models with dense layers. This approach enables WSNs to deliver reliable and efficient real-time data transmission, allowing them to dynamically adapt and optimize transmission schedules in response to changing network conditions.

Figure 4 illustrates the BiLSTM architecture for WSN-DQL. The architecture comprises several layers and operations aimed at extracting meaningful features from the input data and facilitating decision-making in real-time data transmission scheduling. Initially, three inputs (IP-1, IP-2, IP-3) are fed into BiLSTM layers (Bi-1, Bi-2, Bi-3) separately, each with a size of 128. BiLSTM layers are capable of capturing both forward and backward temporal dependencies in the input sequences, making them well-suited for sequential data processing tasks in WSN-DQL. Subsequently, three attention models (AT1, AT2, AT3) are applied to obtain weight matrices, quantity, attention weight, and each time series attention matrix, which help to emphasize relevant features and suppress noise in the input data.

After applying attention models, Avgpool (AP) is individually applied to each attention model output (AT1, AT2, AT3). The resulting values from AP1 and AP2 are concatenated (C1) and passed through a dense layer (C1_D1) with a size of 128. Similarly, the output from AP3 is fed into another dense layer (Bi-3_D2). Following the dense layers, addition (A) and subtraction (S) operations are performed separately on the outputs of dense layers C1_D1 and Bi-3_D2. These operations allow for combining and comparing the features extracted from different parts of the input data.

Next, dense layers (D3 and D4) with a size of 64 are applied to the results of addition (A) and subtraction (S). The outputs from these dense layers are concatenated (C2), followed by additional dense layers (D5, D6, D7) with sizes 64, 32, and 2, respectively. These dense layers help in further feature extraction and dimensionality reduction, enabling the model to capture essential information for decision-making. Finally, a softmax activation function is employs on the output of dense layer (D7) to produce the final outcome of WSN-DQL-BiLSTM, which represents the probabilities of different actions or decisions in the context of real-time data transmission scheduling in WSNs.

The operations described above can be represented as follows: (15)${\displaystyle {Bi}_{ou{t}_i}=Bi\left({IP}_i\right)}$ (16)${\displaystyle {AT}_i=AT\left({Bi}_{ou{t}_i}\right)}$ (17)${\displaystyle {AP}_i=AP\left({AT}_i\right)}$ (18)${\displaystyle C_{1D_1}=ense\left(Concat\left({AP}_1,{AP}_2\right),128\right)}$ (19)${\displaystyle {Bi}_{3_{D_2}}=Dense\left({AP}_3,128\right)}$ (20)${\displaystyle A_{D_1}=\left(C_{1_{D_1}}+B{i}_{3_{D_2}}\right)}$ (21)${\displaystyle S_{D_2}=\left(C_{1_{D_1}}-{Bi}_{3_{D_2}}\right)}$ (22)${\displaystyle D_{Ou{t}_3}=Dense\left(A_{D_1},64\right)}$ (23)${\displaystyle D_{Ou{t}_4}=Dense\left(S_{D_2},64\right)}$ (24)${\displaystyle C_2=Concat\left(D_{Ou{t}_3}\&D_{Ou{t}_4}\right)}$ (25)${\displaystyle D_{Ou{t}_5}=Dense\left(C_2,64\right)}$ (26)${\displaystyle D_{Ou{t}_6}=Dense\left(D_{Ou{t}_5},32\right)}$ (27)${\displaystyle D_{Ou{t}_7}=Dense\left(D_{Ou{t}_6},32\right)}$ (28)${\displaystyle {Final}_{Outcome}=Softmax\left(D_{Ou{t}_7}\right).}$

This sophisticated architecture aims to leverage the capabilities of BiLSTM and attention mechanisms for robust learning and decision-making in Wireless Sensor Networks, providing a comprehensive framework for real-time data transmission scheduling in dynamic and resource-constrained environments.