Wi-Fi sensing gesture control algorithm based on semi-supervised generative adversarial network

Data collection

The sender used a PC equipped with an Intel 5,300 wireless network card to send Wi-Fi signals, and the receiver collected the CSI data, including gestures from another PC equipped with an Intel 5,300 wireless network card through the CSI Tool (Feng et al., 2019). The height of the antenna from the ground is 1 m, and the distance between the antennas is about 5 m. The tested person sits in the middle area of the two computers. In the detection area, the tested person performs five kinds of wave actions, namely, “left”, “right”, “up”, “down”, and “draw a circle”, respectively, and collects 10 s in 4 scenes, in which the gesture action is about 2 s. Each action is repeated 100 times as a data set. The working frequency band of the router can be 2.4 and 5 GHz, respectively. Considering that many devices utilize the 2.4 GHz band and the noise interference is too large, the 5G working frequency band is selected in this experiment.

The article presents a wireless signal sensing system that detects a Wi-Fi signal’s channel bandwidth of 20 MHz. The system uses two subcarriers as its basic group unit and obtains information about the channel state of the 30 channels from each packet.

Preprocessing of raw CSI data

The CSI scores provided in commercial Wi-Fi network cards contain inherent noise. In real life, gesture speed is generally no more than 1 m/s. Therefore, for Wi-Fi equipment working at 5 GHz, the frequency of the CSI’s amplitude waveform change caused by human behavior is f = 2v/λ ≈ 40 Hz. The pertinent data is predominantly found at the lower frequency range of the spectrum, with noise typically occurring at higher frequencies. In such scenarios, the Butterworth low-pass filter is considered more suitable for noise reduction than alternative filters. This filter is advantageous as it minimally distorts the phase information within the signal and provides a maximally flat amplitude response in the passband, thereby avoiding the excessive distortion of a gesture motion signal (Halperin et al., 2011). Consequently, this research employs the Butterworth low-pass filter to denoise the original CSI data initially.

The time series form of the CSI in each different subcarrier caused by human behavior is correlated when each transmission-reception antenna pair is considered. The PCA adopted by the system takes advantage of the correlation to perform secondary denoising action, removing uncorrelated noise components that conventional low-pass filters cannot remove in the signal, which results in pure CSI time series data and improves the accuracy of gesture recognitions. The final data has a reduced dimension and less computational complexity. Thus, the first principal component is selected as the preprocessed CSI data due to having the main and consistent power changes caused by the target motion (Qian et al., 2017).

Feature extraction in the conventional time domain

A feature represents some prominent properties of a sample and is the key to distinguishing gestures. It can be determined whether a sample belongs to a certain class by judging the features extracted from it when classifying or recognizing the sample. After data preprocessing and gesture segmentation, features were extracted from a CSI amplitude as feature vectors for gesture recognition. The manuscript selects eight time-domain characteristics including standard deviation, motion period, median absolute deviation, quartile distance, information entropy, maximum, minimum, and difference score.

Although features can reflect the information of a sample’s properties, different features play distinct roles in classification, and some features may also be redundant, such as the association among the maximum, minimum, and range. Feature selection evaluates the size of the information of different features by a feature evaluation index, removing redundant and negative features from the feature set. The feature selection process reduces the number of attributes but improves the classification performance. After the feature selection process, the classification model also has better generalization and robustness and saves a lot of computational resources and time.

The manuscript adopts the Sequential Forward Selection (SFS) algorithm and Sequential Backward Selection (SBS) algorithm to efficiently screen the subsets of the candidate optimal feature parameters, improving the generalization capability of the classification models.

The SFS algorithm starts from an empty set and adds features X⁺ sequentially so that when combined with the already selected features Y_k, the target function J(Y_k + X⁺) is optimal.

The SBS algorithm begins with the complete set and removes features X⁻ sequentially so that when combined with the already selected features Y_k, the target function J(Y_k − X⁻) is optimal.

From CSI to GASF diagram

Conventional time-domain features will lose a lot of signal information so that CSI signals can be converted into GAF graphs for subsequent signal processing (Boukhechba et al., 2023). The Gramian Angle Field (GAF) method is composed of two different types: the Gramian Angle Sum Field (GASF) and the Gramian Angle Difference Field (GADF). The GASF and GADF images are obtained by computing the cosine function of two angle sums and the sine function of two angle differences, respectively. The GASF image retains more CSI information. Before converting the CSI to a polar coordinate system, it is necessary to scale the CSI to the (0 1) interval employing min-max normalization.

Since the GASF is computed by employing the Angle θ in the polar diagram, H_i should be represented by the corresponding polar coordinate point. The corresponding angle is computed by

(8) $${\theta }_{\mathrm{i}}=\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{c}\mathrm{o}\mathrm{s}({\hat{H}}_i),0\le {\hat{H}}_i\le 1.$$

Therefore, the CSI is converted into images for subsequent processing by utilizing the GASF.

Machine learning classification algorithms

Linear discriminant analysis (LDA) recognizes gesture actions (Breunig et al., 2000) as a typical pattern recognition method belonging to a group of linear classifiers. The core idea is to map multiple categories of data sets to an optimal discriminant vector space to extract category information and effectively compress the spatial dimension of features. This method minimizes the distance between classes, which means that the separability of the model is the best in the whole subdomain. This criterion is also known as Fisher’s differential treatment principle.

The Light Gradient Boosting Machine (LightGBM) employing gradient-boosted decision trees expedites training duration and diminishes memory consumption (Guo, Xu & Chen, 2019). Thus, the performance is enhanced. The LightGBM employs a series of decision trees for sample classification, wherein the gradient of the preceding decision tree is addressed and reduced by the subsequent decision tree until the residual reaches a minimal threshold, thereby achieving an optimal solution. The classification process of the LightGBM is delineated as follows: Let F_m be the initial model, h_m be the base estimator, and Y = F is predicted by minimizing the predefined loss function L (Y, F), where Y and F denote the expected output function and the predicted output function, respectively. The initial model F_m is fitted to the basis estimator h_m so that F_m + h_m = Y and residual h_m = Y − F_m is calculated. The residual gradient of the next subsequent estimator h_m+1 and the previous estimator is fitted so that F_m+1 = F_m + h_m+1 and a new model F_m+1 is constructed. Iterations are continued until the loss function L (Y, F) is minimized and the optimal Y = F is reached. The LightGBM selectively focuses on data instances by exhibiting significant gradients to estimate the information gain. This approach attains precise information gain by using a limited dataset, thereby diminishing time and memory complexities.

The support vector machine (SVM) is run utilizing the LIBSVM software package developed by Ke et al. (2017) for classification. The LIBSVM is recognized for its simplicity, speed, and efficiency in pattern recognition and regression tasks. The software offers a broad range of default parameters and cross-validation functions for SVM, minimizing the need for extensive parameter tuning.

The LIBSVM selects the radial basis function (RBF) as a kernel function, and its decision function is delineated by

(9) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}{\mathrm{t}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}=\mathrm{s}\mathrm{g}\mathrm{n}\left(\sum _{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{w}}_{\mathrm{i}}{\mathrm{e}}^{-\mathrm{g}\mathrm{a}\mathrm{m}\mathrm{m}\mathrm{a}||{\mathrm{x}}_{\mathrm{i}}-\mathrm{x}|{|}^2}+\mathrm{b}\right)$$where ${\text{e}}^{(-\text{gamma}||{\text{x}}_{\text{i}}-\text{x}|{|}^2)}$ represents the kernel function used by the SVM, x_i denotes the support vector in the training dataset, x represents the support vector sample of the predicted label, w_i shows the coefficient of the support vector in the decision function, b denotes the opposite of the constant term in the decision function, and $\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}{\mathrm{t}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}$ represents the decision result. Hence, the system must ascertain the parameters of the decision function to deduce the label of the sample data under evaluation, thereby constructing the current action status. The precision of the system’s recognition is contingent upon the selection of decision function parameters, thereby giving rise to the need for parameter optimization.

Semi-supervised adversarial network generation

Considering the complexity and difficulty of collecting a large amount of CSI data, achieving a high gesture recognition accuracy is particularly significant, even when only a small amount of labelled data exists. Therefore, a semi-supervised GAN is proposed to train gesture classification. The network has the following characteristics: First, it can use many unlabeled data to extract gesture features and improve classification accuracy. Second, it can generate large data through the generation network to enhance the classification model’s generalization capability.

The current methods usually carry out semi-supervised training by adjusting the discriminator structure. However, these methods do not fully consider the potential improvement of the generated data on the classification performance and neglect to enhance the generation performance when the improvement of the model’s fitting capability is considered. Thus, the article proposes an improved semi-supervised GAN based on the Margin-GAN heuristic.

The network structure consists of three main sections: classifier C, generator G, and discriminator D, as shown in Fig. 4. The discriminator serves the same purpose as in a standard GAN, distinguishing whether the sample is from an accurate distribution or produced by a generator. The classifier is trained to increase the classification boundaries of actual samples (both labelled and unlabeled) while simultaneously decreasing the boundaries of the generated fake samples. The generator aims to produce counterfeit samples that look realistic and have significant boundaries, designed to fool both the discriminator and the classifier.

The generator contains two fully connected blocks and four convolution blocks to generate high-quality data. Each convolutional block consists of one deconvolution layer, one batch normalization layer (BNL), and one activation layer. The first three activation functions are modified linear units (ReLU), and the final output activation functions and Tanh functions, respectively. The convolution kernel size is assigned to 4 × 4, the step size is set to 2, and the input is randomly generated noise.

The article constructs a CNN classifier with four convolution layers to extract multi-scale GAF image features. The classifier consists of four convolution blocks, each followed by a batch normalization layer (BNL), which improves the convergence speed. The first two convolution blocks include one convolution layer, 1 BNL, one activation layer, and one maximum pooling layer, and the last convolution block contains only the convolution layer and the activation layer. The convolution kernel size is assigned to 4 × 4, the step size is set to 1, and the pooling layer size is set to 2, activated using the Leaky ReLU function. A dropout layer is added after the third convolutional block to reduce overfitting, and the classification results are output through the softmax function.

The discriminator consists of two convolution blocks and two fully connected layers. Each convolution block contains a convolution layer, a BNL, and an activation layer. The convolution kernel size is assigned to 4 × 4, the step size is set to 2, and the activation function is also used as the Leaky ReLU function.

Performance analysis of different gesture interception algorithms

The article uses the LOF outlier detection algorithm, fixed double-threshold gesture interception algorithm, and dynamic double-threshold gesture interception algorithm to intercept and evaluate gesture movements. The accuracy of gesture interception is shown in Fig. 10.

The evaluation criterion of recognition accuracy was the number of accurately intercepted gestures divided by the total number of gestures.

Figure 10 depicts the accuracies of the LOF algorithm, fixed double threshold interception algorithm, and dynamic double threshold interception algorithm, whose scores are 81.40%, 94.10%, and 98.2%, respectively. The accuracy of the dynamic double threshold interception algorithm exceeds the other two algorithms, which indicates that the algorithm proposed in the article makes the gesture recognition effect more accurate and thoroughly verifies its feasibility.

The analysis of cross-domain performance

The data of one environment is regarded as labelled data for different environments, and the data of the other three environments are randomly divided into test sets and unlabeled data with a ratio of 1:4. The test results are shown in Fig. 11. The proposed algorithm can achieve an average accuracy of 95.67%. Environment 1 achieved the highest accuracy rate of 96.67%. Even the most complex environment 3 had 94.51% accuracy. The implemented label data comprises only 1/3 of the total data implemented in the supervised algorithm. Still, the accuracy of the supervised algorithm is 96.10%, which is consistent with the performance of the proposed algorithm.

The experiments suggest that the proposed algorithm can achieve high gesture classification accuracy even with a small amount of labelled data and a large amount of unlabeled data. The sampling results of the generated data by the generator show that the generated samples cover a variety of gestures, which can effectively represent distinct characteristics of gestures, which is conducive to the training of classification algorithm, and is suitable for scenarios whose characteristics are cross-domain and small sample and needs a fast training stage.

Performance and complexity analysis of the different machine learning algorithms

Table 1 suggests that the performance of the SVM is much better than the LDA. The main reason is that the structure of the LDA is relatively simple, and linear classification is achieved by projecting high-dimensional samples onto the low-dimensional plane. The training operation is fast but has a poor performance. The algorithm complexity is represented by $O\left(d^2\ast n\right)$, where d and n represent the sample dimension and the number of samples, respectively. The SVM maps discrete hyperplanes to a surface based on kernel function, which can effectively overcome the problems of conventional algorithms in the learning process. The trained model has good performance and high recognition accuracy, but the training speed is slower than the LDA. The algorithm complexity is denoted by $O\left(d^2\ast n^2\right)$, where d and n represent the sample dimension and the number of samples, respectively. In future scenarios, the SVM will be more reliable and practical, sacrificing a small amount of training time, but can be exchanged for better performance.

Moreover, the optimization model can be further trained during the non-use period without affecting the user experience. The mean gesture accuracy of the LightGBM can also reach 92.67% and 95.00%, which is similar to the accuracy of the SVM. The LightGBM’s complexity is represented by $O\left(d\ast n\right)$, where d and n represent the sample dimension and the number of samples, respectively. The LightGBM adopts a unilateral gradient algorithm to filter out the samples with low gradients in the training process, which reduces the calculation burden. It also is efficient in faster training. Compared with the SVM, its accuracy is slightly decreased because the number of data sets is insufficient, and a certain degree of accuracy is sacrificed for the training speed.

The classifier algorithms, SVM, LightGBM, and KNN, have different implementation scenarios. For example, the SVM and LightGBM are suitable for linearly separable data or fit nearly linearly separable data well, as well as problems that can be converted to linearly separable by kernel techniques. On the other hand, the KNN is suitable for small data sets and simple class decision boundaries. Therefore, more suitable classifiers are chosen based on the data characteristics.

Therefore, the SVM is suitable for scenarios that require high reliability and are relatively fixed data, such as the fixed setting of a dedicated sensing device in the same scenario. At the same time, the LightGBM is suitable for scenarios that require a fast application and good recognition effect, and it can train an available and reliable model in time.

The examination of the performance analysis before and after the feature selection process

Compared with the algorithm not applying the feature selection procedure, the recognition accuracy of the conventional feature extraction procedure in the time domain is somewhat improved. The feature selection can effectively improve the robustness and generalization of the algorithm.

Figures 12 and 13 show the confusion matrix generated by the SVM before and after the feature selection process. The misrecognition is mainly caused by two gestures, “right” and “down”, since they share certain similarities in the waveforms.

The proposed detection algorithm fits indoor and outdoor environments. However, the multipath effect of the indoor environment is more prominent, and indoor environment interference is less. Therefore, the accuracy of recognition in an indoor environment will be better.

Most researchers employ gestures to realize intelligent non-contact gesture control systems, which need motion-sensing devices similar to Kinect to optimize the real-time image processing algorithm. The Kinect camera cover is implemented to investigate the specific movement changes of the human body and finally realize the control of smart homes according to human movements. However, the gesture control system based on Kinect is accompanied by the data leakage of users’ private information and the inconvenience of wearing sensors. In contrast, the gesture control system implements Wi-Fi signals to recognize gesture actions to control smart homes without collecting any private information from users, providing a new idea for the research to promote the development of “smart homes”.

Multi-person interference

Other people passing by in the perceptual area will cause additional interference to the CSI dynamics collected by the receiver when subjects’ gestures are studied. Thus, gesture recognition is poorly affected. For the proposed perception system, static objects in a sensing region will not affect the gesture information contained in CSI. However, for other people not related to the subject in the sensing area in the real-world scene, the generated interference signals can also be mixed into the CSI dynamics collected by the receiver. Because of the intrinsic issues with Wi-Fi devices, multi-person gesture recognition will still be challenging.

Lightweight model and device deployment

The experimental equipment consists of two notebooks equipped with wireless network cards. In future practical applications, miniaturized devices such as smartphones and wearable devices can be designed since they are easy to deploy and implement. Meanwhile, signals from multiple sources, such as Bluetooth, can be employed to enhance recognition accuracy. To consider using a lightweight algorithm to fit resource-limited end devices, a model compression technology is planned to be employed to decrease the number of parameters and enable real-time inference on terminal devices in the future.

Real-time data learning

In the cross-domain scenario, the commonality of the pre-trained model will be reduced, and the model needs to update its parameters according to the current CSI data to adapt to gesture recognition in distinct environments and improve recognition accuracy. Therefore, a real-time online learning algorithm can be developed in the future so that it can be continuously fed from the real-time data stream and the algorithm parameters can be constantly updated according to the specific application scenarios.