Yoga pose recognition using dual structure convolutional neural network

Motivation

To address the recognition of yoga postures, this work designed a dual structure convolutional neural network with a feature fusion function. On the one hand, we sufficiently borrow the ascendency of convolutional neural networks for feature extraction on images. With the assistance of this ascendency, the global feature and the local feature of a yoga image can be extracted. Hence, we take a dual structure convolutional neural network consisting of the structure CNN A (convolutional neural network A) and the structure CNN B (convolutional neural network B). On the other hand, using the proposed feature fusion function to fuse the extracted global feature and local feature, by doing so, the details at different levels of a yoga image can be focused on, which is beneficial for the recognition of yoga poses.

Contributions

The contributions of this work are summarized.

• (1)

  A dual structure convolutional neural network is used for the recognition of yoga postures, where the structure CNN A can observe different channels of finding the global features, and the structure CNN B can calculate the depth information in each pixel of an image. Together, the two structures pay attention to critical features of yoga images.

• (2)

  The proposed feature fusion function is proved to have positive effects on the recognition of yoga postures. Moreover, the feature fusion with a matrix dot multiplication operation can yield more effectiveness in the improvement of the recognition for yoga postures than that with a direct connection operation does. The values of feature fusion can afford the direction of ongoing yoga and human posture.

This work thus is arranged below as follows. The following section, the related literature is summarized in Related Works. Methodology discusses the proposed model architecture and the proposed feature fusion function. Experiments and Results puts a discussion based on the experimental analytics. Lastly, Conclusion draws this work and directs future works.

Model’s structure

Figure 1 displays the structure of DSCNN, including the structure CNN A and the structure CNN B. Because the two structures CNN A and CCN B focus on different objectives, CCN A adopts the two-layer structure, while CNN B adopts the three-layer structure. The details are as follows.

The structure CNN A is used to focus on the global feature of a yoga image. To extract the global feature at coarse granularity, in structure CNN A, we consider a double layer structure, including $50\times 50\times 16$ convolution layer, batch normalization and rectified linear unit (ReLU), followed by $25\times 25\times 128$ convolution layer and a MaxPooling layer.

The structure CNN B is to finely extract the local feature of a yoga image, which can pay more attention to the details of a yoga pose, i.e., the so-called ‘local feature’. In structure CNN B, we designed three convolutional layers, including $50\times 50\times 32$ convolution layer, batch normalization and ReLU, followed by $50\times 50\times 64$ convolution layer, batch normalization, ReLU and a MaxPooling layer. Thereafter, $25\times 25\times 128$ convolution layer, batch normalization ReLU, and a MaxPooling layer. After the global feature and the local feature are integrated, they enter a fully connected layer. Finally, the recognition of yoga poses is implemented in the Softmax layer.

Here, the global feature extraction and the local feature extraction have different purposes. For example, like the two yoga poses in the literature (Fig. 5C in Verma et al., 2020), when the two yoga poses are the same, but there are obvious differences in the background, to prevent misrecognition caused by background interference, hence, we considered the global feature. That is, the role of the global feature is to resist the interference of the background in an image. However, when the two yoga poses have obvious differences, in this scenario, the local feature can directly recognize yoga poses. Although our goal is to recognize yoga poses of input images, the background of input images may create interference. Accordingly, we took into account the global feature to resist the interference. Certainly, the recognition of yoga poses mainly relies on the local feature. Overall, the roles of the local feature and the global feature are different.

Feature fusion

This subsection starts to illustrate how to fuse the global feature and the local feature. For the convenience of description, let us assume the global feature $F_{\mathrm{g}}=\left[\begin{array}{cccc}{g}_{11}& g_{12}& ...& g_{1m}\\ g_{21}& g_{22}& ...& g_{2m}\\ ...& ...& ...& ...\\ g_{m1}& g_{m2}& ...& g_{mm}\end{array}\right]$ and the local feature $F_l=\left[\begin{array}{cccc}{l}_{11}& l_{12}& ...& l_{1m}\\ l_{21}& l_{22}& ...& l_{2m}\\ ...& ...& ...& ...\\ l_{m1}& l_{m2}& ...& l_{mm}\end{array}\right]$.

The feature fusion function $\mathrm{\Omega }$ is given in Eq. (1).

(1) $$\mathrm{\Omega }=(F_g\ast \alpha )\otimes (F_l\ast (1-\alpha ))$$where symbol $\otimes$ represents a matrix dot multiplication operation. Item $\alpha$ is a weight coefficient and the calculation is below. The mean of $F_{\mathrm{g}}$ and $F_l$ is calculated as follows,

(2) $$\{\begin{array}{c}u(F_g)=\frac{1}{m}\sum _{i,j}^m{g}_{ij}\hfill \\ u(F_l)=\frac{1}{m}\sum _{i,j}^m{l}_{ij}\hfill \end{array}.$$

The corresponding standard variance is calculated in Eq. (3)

(3) $$\{\begin{array}{c}\sigma (F_g)={\left({\displaystyle \frac{1}{m}\sum _{i,j}^m{\left(g_{ij}-u(F_g)\right)}^2}\right)}^{{\displaystyle \frac{1}{2}}}\hfill \\ \sigma (F_l)={\left({\displaystyle \frac{1}{m}\sum _{i,j}^m{\left(l_{ij}-u(F_l)\right)}^2}\right)}^{{\displaystyle \frac{1}{2}}}\hfill \end{array}.$$

Now, let us calculate the item $\alpha$. As follows

(4) $$\alpha =\{\begin{array}{cc}{\beta }_1,& \mathrm{i}\mathrm{f}{\beta }_1<{\beta }_2\hfill \\ {\beta }_2,& \mathrm{i}\mathrm{f}{\beta }_2<{\beta }_1\hfill \end{array}$$

(5) $$\{\begin{array}{c}{\beta }_1=\frac{{\mathrm{\Delta }}_1}{{\mathrm{\Delta }}_1+{\mathrm{\Delta }}_2}\hfill \\ {\beta }_2=\frac{{\mathrm{\Delta }}_2}{{\mathrm{\Delta }}_1+{\mathrm{\Delta }}_2}\hfill \\ {\mathrm{\Delta }}_1=\frac{u(F_g)}{{\left|{\sigma }^2(F_g)+{\left(\frac{1}{m}\sum _{i,j}^m{\left(g_{ij}-u(F_g)\right)}^3\right)}^{\frac{1}{3}}\right|}^{\frac{1}{2}}}\hfill \\ {\mathrm{\Delta }}_2=\frac{u(F_l)}{{\left|{\sigma }^2(F_l)+{\left(\frac{1}{m}\sum _{i,j}^m{\left(l_{ij}-u(F_l)\right)}^3\right)}^{\frac{1}{3}}\right|}^{\frac{1}{2}}}\hfill \end{array}$$where items ${\left({\displaystyle \frac{1}{m}\sum _{i,j}^m{\left(g_{ij}-u(F_g)\right)}^3}\right)}^{\frac{1}{3}}$ and ${\left({\displaystyle \frac{1}{m}\sum _{i,j}^m{\left(l_{ij}-u(F_l)\right)}^3}\right)}^{\frac{1}{3}}$ are to increase the weight ratio between the global feature and the local feature, thus allowing the local feature to obtain higher weights.

Model parameters

• Convolutional kernel. The main purpose of the convolution operation is to achieve feature extraction of an image. The Convolutional kernel includes High Pass Filter (HPF) and Low Pass Filter (LPF), among them, HPF means that the high-frequency part of an image (i.e., the part with more dramatic changes in an image) is allowed to pass through. HPF is often used to sharpen an image and enhance the edges of objects in the image. Such as Sobel operator, Prewitt operator, sharpening filter, etc. However, LPF implies that the low-frequency part of an image (i.e., the part with the slowest change in an image) is allowed to pass through. LPF is often used to blur or smooth an image, eliminate noise, etc. Such as Gaussian filter, mean filter, etc. In terms of our DSCNN, we chose a 3 × 3 4-adjacent Laplace operator. Because the edges in an image are those areas where the grayscale changes, the Laplace operator is very useful in edge detection.

• Max pooling. During the pooling process, max pooling only retains the maximum value of each region, ensuring that the strongest features can be retained and other weak features are discarded. This can assist the improvement of recognition ability for the model to key features. Therefore, a 2 × 2 max-pooling is used in our DSCNN.

• Batch size. Figure 2 displays the relationship between batch size, training epoch and training accuracy (Smith et al., 2017). It can be seen that the processing speed to the same amount of the data augments with the increase of batch size. Due to the final convergence accuracy falls into different local extremes, the batch size is increased to a certain size to obtain the best final convergence accuracy. Taking Fig. 2 as a reference, the batch size is dynamically adjusted during the training phase.

• Activation function. The most commonly used ReLU activation function is used in our DSCNN. Equation (6) gives the ReLU activation function

(6) $$f(x)=max(0,x)$$The reviewer ReLU function is a piecewise linear function, which sets all negative values to 0, while it leaves positive values unchanged. This operation is called lateral inhibition, because of the lateral inhibition, neurons in neural networks exhibit sparse activation. This is particularly evident in deep neural network models, such as CNNs. When training a deep classification model, there are often only a few features related to the target. Indeed, a sparse model achieved through ReLU can better mine relevant features and fit the training data. Consequently, we chose the ReLU function as our activation function.

Algorithm implementation

The algorithm is as shown in Algorithm 1. The input and output are an image set I_set, training accuracy and testing accuracy, respectively. Firstly, image set I_set is randomly divided into the training set Train_set and testing set Test_set, where 70% of the image set is used for the training set, and the rest is used as the testing set, illustrated in Step 1 to Step 3. Following that, the structures CNN A and CNN B are initialized in Step 4 and Step 5. The process between Step 6 and Step 21 describes model’s training. Structure CNN A extracts global features F_g and Structure CNN B extracts local features F_l, illustrated in Step 7 to Step 10. The extracted global features and local features are fused by the feature fusion function Eq. (1), as shown in Step 11 to Step 14. According to the fused features, Softmax layer recognizes yoga postures in Step 15. If the model starts to converge, the training is terminated. Current training model is saved, and the training accuracy is outputted, as shown in Step 16 to Step 20. Finally, using the testing set Test_set to test the trained model, and the testing accuracy is outputted, illustrated in Step 22 and Step 23.

Experimental datasets

The experimental dataset is cited from the Yoga-82 dataset in Verma et al. (2020). We chose five types of yoga poses from the Yoga-82 dataset, including Standing yoga pose, Twisting yoga pose, Forward Bend yoga pose, Kundalini yoga pose and Prenatal yoga pose. (Regarding the four yoga pose and the Yoga-82 dataset, please refer https://sites.google.com/view/yoga-82/home). The experimental dataset consists of 280 images, of which 70% is used for the training set, and the rest 30% for validation set.

To evaluate our DSCNN, we selected five competitors Resnet 50 (He et al., 2016), Inception V3 (Szegedy et al., 2016), Xception (Chollet, 2017), Inception-Resnet-V2 (Szegedy et al., 2017), YoNet (Ashraf et al., 2023). Then, using the four metrics Accuracy, Precision, Recall, F1-score to evaluate their recognition ability to yoga poses. Unless otherwise stated, our model and the five competitors are running in the same experimental environment.

Experimental designs

To verify the proposed method, multiple experiments were designed.

• (i)

  To verify the effectiveness of the feature fusion function, the ablation experiments were carried out. Specifically, four manners were designed to recognize the five types of yoga poses. The first one is that we singly use the structure CNN A in Fig. 1 to recognize yoga poses, and in the second manner, using the structure CNN B in Fig. 1 to identify yoga poses. As for the third manner, denoted as CNN A + CNN B, the local feature extracted by CNN B and the global feature extracted by CNN A are directly connected, rather than using the feature fusion function $\mathrm{\Omega }$, i.e., the global feature is directly followed by the local feature. While for the fourth manner, using the feature fusion function $\mathrm{\Omega }$, i.e., using our DSCNN to recognize the five types of yoga poses.

• (ii)

  To verify recognition performance, we compared the DSCNN and the five competitors Resnet 50, Inception V3, Xception, Inception-Resnet-V2, YoNet. Using the four evaluated metrics Accuracy, Precision, Recall, F1-score to analyze the recognition results.

Result analysis

This subsection displayed the experimental results, which contains the validation on feature fusion and the recognition ability to the five types of yoga poses. Experimental results imply that our DSCNN defeated against the five competitors in recognition performance to the five types of yoga poses, and achieved 97.23% accuracy with 96.08% precision, The detailed results are illustrated as follows.

(i) Validation on feature fusion

To verify the feature fusion function $\mathrm{\Omega }$ in Eq. (1), the ablation experiment was designed. The results of the ablation experiment are illustrated in Table 1. Through observing the four evaluation metrics accuracy, precision, recall, F1-score, we find that the best results are obtained by DSCNN, i.e., the fourth manner. The won advantages of DSCNN imply that the feature fusion function $\mathrm{\Omega }$ successfully fuses the extracted global features and the extracted local features, so that the recognition performance of the DSCNN is significantly augmented. Together, these results in the ablation experiment confirm that the proposed feature fusion function can assist the improvement of the recognition ability of the model.

(ii) Comparisons of recognition performance

This subsection compared our DSCNN against the five competitors Resnet 50, Inception V3, Xception, Inception-Resnet-V2, YoNet in terms of the recognition of the five types of yoga poses. Compared results in Table 2 show that our DSCNN defeats against the five competitors. In terms of the evaluated metric accuracy, our DSCNN is 97.23%, which is superior to the five competitors. However, the competitor Inception-Resnet-V2 obtains the lowest accuracy. Observing the three evaluation metrics precision, recall, F1-score, our DSCNN also shows significant advantages.

To test the statistical significance of the difference between the six algorithms, the Wilcoxon-test was adopted. Average ranks of the six algorithms are calculated by using $({\sum }_{i=1}^6{r}_i^j)/6$, where $r_i^j$ is the ranking of i-th algorithm on j-th dataset. The tested results in Table 2 show that our DSCNN obtaining the best average ranks is statistically better than the five competitors at the 95% confidence level. Furthermore, there are no differences between them for these comparative results.

Additionally, the fundamental target of our DSCNN can fuse the global feature and the local feature of an image to implement a recognition decision. For that purpose, we explore the feature map for both merged types of convolutions and visualize them, illustrated in Fig. 3. For the yoga poses—“Twisting”, Fig. 3A displays the feature map in the third convolutional layer of the structure CNN A where we can observe different channels finding the global features, such as edges. Figure 3B displays the feature map in the third convolutional layer of the structure CNN B that calculates the depth information in each pixel of the image. Figure 3C displays the fused feature map in the third convolutional layer of our DSCNN.

To further analyze the recognized results, Fig. 4 displays the confusion matrix, showing that the recognition accuracy for the Prenatal yoga poses is lower, while the recognition accuracy on other yoga poses is higher. Additionally, Fig. 5 displays the loss in process of training and validation. Through observation, our DSCNN can converge when the training reaches 110 epochs, and the loss value is 0.44. Moreover, there was no significant vibration in the loss value during the training process, indicating that our DSCNN has certain robustness.

Advantages

The proposed method outperforms the comparative methods, and the reason can be interpreted as follows. To accurately recognize the five different yoga poses, we considered the global features and local features. This is to pay attention to the five postures from different perspectives. Then, using the feature fusion function Eq. (1) to fuse the extracted global features and local features. In fact, the feature fusion function Eq. (1) is a key factor in our success since it assigns higher weights to the local feature, thereby allowing our model to pay attention to the differences in the five different yoga postures. That is why the proposed method won over the competitors.

Limitations

For the recognition of yoga postures, our method relied on the feature fusion function Eq. (1). Together, the quality of both the extracted global features and local features determined the calculation accuracy of feature fusion function Eq. (1). Here, this work did not consider noisy interference, that is to say, the anti-noisy ability of the feature function Eq. (1) is not proven. If the local features of yoga postures are masked by noise, the proposed model might obtain a poor recognition accuracy. This does not mean that the proposed model does not have the ability to resist noise. The calculation accuracy of the feature fusion function Eq. (1) may decrease, leading to a decline in our model’s ability to recognize yoga postures.

Insights

However, in practical applications, yoga pose recognition faces various challenges. (i) Limitations of the dataset. Yoga pose recognition relies on a large amount of annotated data for model training. But unfortunately, existing public datasets often lack diversity and cannot cover all types of yoga poses and different practitioner states. Especially for some complex yoga poses, the sample size in the dataset may be very limited, which limits the model’s generalization ability. (ii) Real time and accuracy. In practical applications, yoga pose recognition requires real-time operation on mobile devices, which places high demands on the computational efficiency of the algorithm. Meanwhile, to ensure the accuracy of recognition, the algorithm needs to be able to handle various lighting conditions, background interference, and changes in human posture. These factors will all affect the accuracy and real-time performance of recognition. (iii) Detection of joints and key points. The core of yoga pose recognition is the detection and localization of key points on the human body. However, the posture changes of human joints are diverse, sometimes the joints may be obstructed, or in some poses, the posture of the joints may be very close, which increases the difficulty of key point detection. In addition, differences between individuals can also affect the accuracy of key point detection.