Hand gesture recognition via deep data optimization and 3D reconstruction

Pre-processing

In this subsection, we cover the pre-processing for the suggested technique, which begins with foreground extraction using change recognition and associated component-based approaches and strategies. The connects a component labelling approach for fragmenting the human hand silhouette and finding conspicuous skin pixels.

We adopt these techniques from different studies. For example, using Otsu’s method and image segmentation, Petro & Pereira (2021) proposed a novel method for optimal color quantization. On numerous test images, author revealed that their method performs better than the other approaches. Lisowska, Tabor & Ogiela (2021) provided a novel image thresholding technique that combines fuzzy c-means clustering and multi-Otsu’s method. The proposed techniques were tested on several benchmark datasets which shows better results. In another study, Zhang & Jia (2021) included a new version of Otsu’s method among their list of novel histogram-oriented thresholding techniques for remote sensing images.

Utilizing histogram-oriented thresholding, we differentiated the hand shape after extracting the skin elements. Using Otsu’s method, numerous threshold standards of Δ (Eq. 1) were adapted, and the extreme color strength of stochastic histogram hso (x,y) is described as: (1)${\displaystyle hiso\left(x,y\right)=\left\{\begin{array}{c}IT\left(\frac{TOi+Teh{i}_{max}}{4}\right),ifTeha{i}_{max}\le 5\hfill \\ IT\left(\frac{TOi+Teh{i}_{max}}{2}\right),ifTeha{i}_{max}>5\hfill \end{array}\right.}$ where IR is the overweight, TOai is a threshold which is suggested by Otsu’s technique, and Tehai_max is the main position of skin occurrence over extracted histogram directory . This process is practical for every grey scale subdivision of given image, which articulated as; (2)${\displaystyle Ige\left(x,y\right)=\left\{\begin{array}{c}\left(0,0,0\right),ifgr\left(x,y\right)=0\hfill \\ \left(Igr\left(x,y\right),Imgr\left(x,y\right),Ib\right),ifg\left(x,y\right)=1\hfill \end{array}\right..}$

Hand detection

The humanoid hand silhouette ridge identification process consists of two steps (Zhang et al., 2018): sequential edge identification and ridge data synthesis. In the binary border separation process, binary boundaries are recovered from the RGB silhouettes produced in the above-mentioned denoising phase. Employing distance transformation, the proximity maps are generated on the boundaries. On the other hand, the ridgeline data creation phase, the local optima are acquired from the pre-estimated mapping to generate ridge data along the binary vertices. The mathematical representation of hand recognition is (3)${\displaystyle \gamma K=n\sum tx=1\left|\left|\alpha x\right|-\left|\beta \right|\right|whereK=1,2,3\dots }$ where α symbolizes the centroid point of the trajectories deposited in the confusion table, β denotes the updated trajectories of the evaluated data, γ indicates the detachment among the present standards of the confusion matrix and the novel trajectories.

Hand points detection

The segmentation hand is then utilized to detect landmarks. Numerous methods are offered for localizing hand landmarks, which aid in extracting the features for recognizing and identifying individual movements. The bulk of strategies are quite straightforward and restrict the precise location of landmarks. After collecting the acoustic waves of geodesic velocity using the fast-marching algorithm (FMA) on frames, landmark recognition is conducted. The color values of quads p are generated based on the outlines’ outer boundaries b. Pixels with identical color values c are identified and then the mean is calculated; based on the pixel’s average values, the landmark l is painted. For the innermost landmark, the value of bright green is determined and the distances adjacent points is determined. Calculating the fingers yields (4)${\displaystyle l=c\left(px,py\right)/2}$ where px and py have the same color in the external surface and cpx, py is the total amount of pixels with that colour in the external surface. On the hand outlines in Fig. 2 below, landmark is inspired:

Features extraction

This step presents the details of features abstraction techniques for hand gesture recognition over challenging databases. We employed three detail methods for acquisition of features: geometric features, 3D points modeling and reconstruction, angular point features. Algorithm 1 provides the complete methodology for features abstraction.

Geometric features

The point-based approach is used to retrieve hand sign features, which comprise locations on the thumb, forefinger, ring finger, ring thumb, or little finger (see Fig. 3). All the values are merged in numerous ways to provide a range of learning and recognition-related properties. These markers are positional, geometrical, and angle-based properties. The proximity attribute d calculates the distance between ixy radical landmark on the fingertips and the cxy interior marker using hand’s geodesic value, which is expressed as (5)${\displaystyle ∥d∥=\sqrt{{\left(x{j}_2-x{b}_1\right)}^2+{\left(y{j}_2-y{b}_1\right)}^2}}$ while d is indeed the distance between both places; xi and xcare the xcoordinates of the hand’s outer reference and inner historic site, congruently; and yc and yi are the y organizes of the identical structures.

3D point modeling and reconstruction

The first stage in this section of the three-dimensional hand reconstructions is accomplished using a mathematical model and ellipsoid approaches. Using the details from the connections, we can see that an ellipsoid connects the hand point to the following point and the index point to the inner elements. The rest of the human hand points are connected to the inner hand in a similar manner to how the thumb position is attached to it using an ellipsoid structure. The Eq. (6) shows the formulation of three-dimensional hand reconstructions and computational model. (6)${\displaystyle k_{me}=l_a\left(e_x,e_y\right)■\left(i_x+1\right)\left(e_x,e_y\right)}$ Where k_me denotes the computational model and l_a(e_x, e_y) is the first points and x,y are the values. Figure 4 shows the detail overview of computational model and 3D hand reconstructions.

Data optimization: Grey Wolf Optimization (GWO)

The GWO algorithm is an intelligent swarm technique created by Rezaei, Bozorg-Haddad & Chu (2018), which imitates the wolf’s governing system for cooperative exploration. The grey wolf is a member of the Canidae family and enjoys living in packs. Wolves have a strong hierarchy, with a male or female alpha as their commander. The alpha is mainly tasked with making decisions. The package must accept the leader’s instructions. Betas are senior wolves who assist the leader in making decisions. The beta serves as alpha’s consultant and administrator. Omega, the lowest-ranking grey wolf, must notify the majority of other dominating wolves. A wolf is a delta if it’s neither an alpha, beta, or omega. The omega is governed by the delta, which interfaces with the alpha and beta. Wolves’ hunt strategies and social stratification are represented statistically to develop GWO and achieve improvement. The GWO methodology is evaluated using standard test methodologies, which reveal that it is comparable to other swarm-based approaches in terms of identification and application.

When prey is instigated, the reappearance rises (t = 1). Subsequently, the alpha, beta, and delta wolves would supervise the omegas to search and eventually squeeze the prey. Three measures $\bar{A}$, Ź, F and Ŷ are predictable to define the surrounding performance:

${\displaystyle \text{Ŷ}\alpha =\left|\text{Ź}1.\bar{A}\alpha -\bar{A}\left(t\right)\right|,}$ (10)${\displaystyle \text{Ŷ}\beta =\left|\text{Ź}2.\bar{A}\beta -\bar{A}\left(t\right)\right|,}$ ${\displaystyle \hat{Y}\delta =\left|\text{Z}3.\bar{A}\delta -\bar{A}\left(t\right),\right|}$ where t requires the existing repetition, $\bar{A}$ is the position trajectory of the grey wolf, and $\bar{A}1$, $\bar{A}2$ and $\bar{A}$ are the position trajectories of the alpha, beta, and delta wolves. Algorithm 02 shows a comprehensive indication of the data optimization technique via grey wolf optimization.

Classification: ANN

This section involves the classification method via ANN. An ANN is a set of numerous perceptron or neuron on every stratum; when required information is characterized in the forward channel, this is referred to as a feed-forward neural network (Abdolrasol et al., 2021).

Artificial neural networks (ANNs) have the capability to recognize hand gestures and can be trained to solve intricate problems that traditional computing systems or individuals typically encounter. Superintended training methods are frequently employed in practice, although there are instances where unsupervised training techniques or direct design methods are also utilized. As discussed in the literature, an artificial neural network has been utilized to detect gestures (Nguyen, Huynh & Meunier, 2013). The segmentation of images in this system was carried out by utilizing skin color as a basis. The features chosen for the artificial neural network (ANN) comprised adjustments in pixel values across cross-sectional planes, boundary characteristics, and scalar attributes such as aspect ratio and edge ratio.

In addition, the ANN method is effective for handling problems involving RGB data, textual information, and contingency table. The benefit of ANN is its ability to deal with transfer function and its ability to understand variables that translate any inputs to any result for any data. The artificial neurons endow their ANN with substantial qualities that enable the network to understand any complicated relation between output and input data, often known as a universal approximation. Numerous academics use ANNs to tackle intricate relationships, such as the cohabitation of mobile and WiFi connections in spectrum resources. We pass the important features vector to neural network for classification; Fig. 5 shows the model diagram of ANN.

Validation methods

The LOSO CV strategy has been adopted to evaluate the performance of the HGR framework on two distinct benchmark databases, namely the IPN hand and Jester databases, respectively. The LOSO approach is a derived form of cross-validation that utilizes data from an individual participant for each fold.

Datasets description

The IPN hand dataset (Benitez-Garcia et al., 2021) is a broad-scale video dataset of hand gestures. It includes pointing with one finger, pointing with two fingers, and other complex gestures. The IPN dataset consists 640 × 480 RGB videos at 30 frames per second.

The Jester dataset (Materzynska et al., 2019) contains an extensive number of webcam-collected, tagged visual content of hand motions. Single video sequence is transformed to a jpg frame at a frequency of 12 frames every second. The database includes 148,092 films. There are 27 different categories of hand gestures.

Experimental evaluation

The MATLAB (R2021a) is utilized for all testing and training while Intel (R) Core i5-10210u Quadcore CPU @ 1.6Ghz with x64 Windows 11 was programmed as the primary device. In addition, the device encompassed with an 8GB RAM.

The next stage of this research was to assess the performance of proposed system on two different databases. Therefore, we utilized grey-wolf optimized ANN for classification. Figure 6 represents the 13 hand gestures of confusion matrix of IPN hand database with an classification recognition rate of 89.92%. Figure 7 depicts the confusion matrix of Jester database with a recognition rate of 89.76%.

(Note: H1 = pointing with two fingers, H2 = pointing with one finger, H3 = click with two fingers, H4 = click with one finger, H5 = throw up, H6 = throw down, H7 = throw left, H8 = throw right, H9 = open twice, H10 = double click with two fingers, H11 = double click with one finger, H12 = zoom in, H13 = zoom out).

(Note: J1 = swiping left, J2 = swiping right, J3 = swiping down, J4 = swiping up, J5 = thumb down, J6 = thumb up, J7 = zooming out with full hand, J8 = zooming in with full hand, J9 = rolling hand forward, J10 = stop, J11 = rolling hand backward, J12 = shaking hand, J13 = pulling hand in).

Evaluation with other state-of-the-art algorithms

In this section, we evaluated our system with other classifiers. Additionally, for the classification of HGR, we compared our proposed system with other sophisticated approaches such as AdaBoost and Decision trees. Figure 8 shows the comparison of IPN Hand and Jester databases over state-of-the-art methods.

While the Figs. 9–10 shows the comparison of ANN with AdaBoost and decision trees recognition accuracies over IPN hand dataset. The results shows that Adaboost achieved 86.84% and decision trees attained 84.38% over IPN hand dataset. Therefore, results clearly shows that ANN outperformed both classifiers in terms of recognition accuracy over IPN hand dataset.

In this experiment, the Adaboost achieved 87.46% and decision trees attained 84.23%. Therefore, results clearly shows that ANN outperformed both classifiers in terms of recognition accuracy over Jester dataset.

We also compared our proposed system with other performance metrics including precision, recall, and F-1 score. Table 1 presents the performance metrics results over IPN hand gesture dataset. Table 2 shows the performance metrics results over Jester dataset.