A multimedia art interaction model for motion recognition based on data-driven model

The framework for openpsoe

In human–computer interaction, human motion recognition plays a pivotal role, and extracting human motion features holds significant importance. However, when confronted with scenarios involving an uncertain number of people, occlusion of objects, or occlusion between individuals, these factors can considerably impact the recognition methodology based on image classification for human motion postures. Consequently, to address this challenge, the OpenPose neural network introduces the concept of part affinity fields (PAFs). Essentially, PAFs represent a set of two-dimensional vector fields that contain crucial information about the position and direction of the limbs. The degree of association between these joint points can be calculated by utilizing the positional information and PAFs associated with each joint point of the skeletal structure. This calculation enables the subsequent fusion of the human model, completing the process of assembling a comprehensive representation of the human body and its movements (Shi, Wang & Zhao, 2022).

OpenPose is a two-step framework that can be expressed by the Eqs. (1) and (2) (Qiao, Wang & Jian, 2018): (1)${\displaystyle S^t={\rho }^t\left(F,S^{t-1},L^{t-1}\right)}$ (2)${\displaystyle L^t={\phi }^t\left(F,S^{t-1},L^{t-1}\right).}$

St is the joint point confidence map output, and Lt represents the nodes relationship among different nodes, which can also be named as the weight coefficient. In Eqs. (1) and (2), the time step t are both more than 2. Considering the loss function, L2 loss function was employed for both branches. The loss in S and L can be shown as Eqs. (3) and (4): (3)${\displaystyle {\text{Loss}}_s^t=\sum _{i=1}^J\sum W\left(p\right)•\parallel \underset{i}{\overset{t}{S}}\left(p\right)-S_i^{\ast }\left(p\right){\parallel }_2^2}$ (4)${\displaystyle {\text{Loss}}_L^t=\sum _{j=1}^J\sum W\left(p\right)•\parallel \underset{j}{\overset{t}{L}}\left(p\right)-L_j^{\ast }\left(p\right){\parallel }_2^2}$ (5)${\displaystyle L=\sum _{t=1}^T\left(f_s^t+f_L^t\right)}$ where superscript * is the true value and the one without that represents the predicted value, p represents each pixel, W (p) means the point is missing at this point. Combining the loss from two steps, the overall loss is obtained in Eq. (5).

After the preliminary training of the network, confidence detection of the human body region is required. There are 19 key joint nodes in an image of the human body. For the k person, the confidence map of position j will be generated. xj, k are the positions of the kth individual part j in the figure. It is through Gaussian modeling to obtain $S_{j,k}^{\ast }\left(p\right)$ the thermal diagram of the pixel p in the image with respect to the body part j. The calculation formula is shown in Eq. (6). the maximum value of the response is taken for multiple responses generated. (6)${\displaystyle S_{j,k}^{\ast }\left(p\right)=exp\left(-∥p-x_{j,k}∥/{\sigma }^2\right).}$

On this basis, complete the skeleton node association based on the partial affinity field and realize the bipartite graph matching with the maximum edge weight value to achieve high-precision calculation of human motion posture. The projection relationship between two points in the limb is used to calculate the angle between each joint. The calculation can be represented by the following Eq. (7) as follows: (7)${\displaystyle {\theta }_n=arccos\frac{r_{i,j}^1•r_{i,k}^1+r_{i,j}^2•r_{i,k}^2}{\sqrt{r_{i,j}^{12}+r_{i,j}^{22}}+\sqrt{r_{i,k}^{12}+r_{i,k}^{22}}}.}$

Among them, r_ij represents the vector of the limb joint necklace starting from i to j. θ_n represents the angle between the nth limb and is formed by the shutdown calculation between the two limbs r_ij and r_ik. Therefore, the original video stream data can be extracted through the OpenPose network and joint point data to obtain the data relationships between various joints, forming the angle features of the motion joints.

Key motion feature extraction using wearable devices

The motion information acquisition of wearable devices mainly depends on the inertial sensors carried by the device itself. Inertial devices applied to wearable devices generally have low accuracy at low cost, and their accuracy is limited. Only time-domain signals are analyzed according to the actual situation of the collected signals. In this article, the sliding average method of the most used digital filtering method is adopted, and its calculation method is shown in Eq. (8) (Wang et al., 2018) (8)${\displaystyle p_t=\frac{\sum _{i=1}^n\left(x_{t-i}+x_{t+i}\right)+x_t}{2n+1}.}$

The window size directly affects the filtering result. If the window is too large, the filtering result will be smoother. However, the small window size may deviate from the true value to some extent and the large window may induce more noise.

As data filtered, the feature representing the motion should be extracted. In this article, the traditional statisticals like the mean, extre and skewness are all calculated. At the same time, the motion variance and intensity based on the sliding window is also employed as shown Eqs. (9) and (10) (9)${\displaystyle {\sigma }_{a_i}^2=\frac{1}{2w+1}\sum _{j=i-w}^{j=i+w}{\left(a_j-{\overline{a}}_j\right)}^2}$ (10)${\displaystyle M_{\text{in}}=\sqrt{a_x^2+a_y^2+a_z^2}}$ where ${\sigma }_{a_i}^2$ represents the sliding variance results of each axis of the three-axis acceleration, aj is the heart rate value at the j, w is the sliding window’s width, and the mean value in the j point window. Equation (9) can be obtained by calculating the square sum of the three-axis accelerations in the inertial sensor, which can reflect the intensity change of motion with high accuracy to a certain extent (Peng et al., 2021).

Motion recognition based on the data-driving model

The data-driven model can be close to the law of data according to the change of data itself; that is, the model can adapt to the situation and change according to different data changes. Most of the traditional machine learning methods belong to this category, and the model training is completed by collecting data. However, in multimedia human–computer interaction, all kinds of actions are often not independent. So, it is necessary to refer to the actions in the previous stage, that is, consider the continuity of the time series to complete. Therefore, although traditional machine learning algorithms such as K-nearest neighbor (KNN), support vector machine (SVM), C4.5 (one of the decision tree methods), etc., can achieve classification in time step, they do not have time consistency. Estimating the previous state is difficult, which greatly reduces the experimental results. Therefore, the cyclic neural network is needed for the sequence with time continuity. The mathematical expressions of traditional neural networks and recurrent neural networks (RNN) can be expressed by Eqs. (11) and (12), respectively (Wagstaff & Kelly, 2018): (11)${\displaystyle h_t=\sigma \left({\omega }_{x_t}{x}_t+b\right)}$ (12)${\displaystyle h_t=\sigma \left({\omega }_{x_t}{x}_t+{\omega }_{h_t}{h}_{t-1}+b\right).}$

It can be found from Eqs. (11) and (12) that recurrent neural network (RNN) adds the input state ht-1 of the previous time in the function input process, which allows it to remember the output state of the previous time, making it an advantage in processing time series. The fundamental characteristic that sets RNNs apart from other neural networks is their ability to maintain a hidden state or memory of past inputs while processing the current input. This hidden state allows RNNs to capture temporal dependencies and learn patterns over time. RNNs use the same set of weights across all time steps. This shared weight structure enables them to learn and recognize patterns in sequential data efficiently. Therefore, this article completes recognizing human motion sequences through the cycle time network RNN.

Human skeleton information

In this article, to meet the interaction needs in multimedia art design, when designing experiments, first, let volunteers wear a bracelet to collect inertial information on their hands, then let them sit on a stool and remain still for a period, and then stand up to complete the lifting of their left hand, right hand and hands in turn. Three volunteers were recruited for the experiment, and each experiment was repeated five times. The data’s safety should also be ensured when capturing the volunteers’ video. Hence, in this article, the video is only used for scientific use and the consent is also acquired from the volunteers (Haider et al., 2021). After capturing the video, the subject-specific training was selected; two subjects were used to train the model; one was the test.

The camera used in the experiment is an RGB webcam. The skeleton information extracted through OpenPose is shown in Fig. 2, which is limited by the height, obesity, and thinness of different people. In addition to referring to Eq. (6), if extracting joint information, the skeleton is adjusted before using it to calculate joint angles to improve the calculation accuracy of each joint angle.

Inertial information from wearable device

For the inertial signal, given the characteristics of the current equipment, this study collected the hand motion information of the experimental object. The 3D acceleration signal of the hand during the experiment is extracted. After extraction, the noise in the movement is removed using mean filtering. One of the simplest ways to gauge exercise intensity is by calculating the magnitude of acceleration. Higher acceleration values generally correspond to more intense physical activity. For example, running or vigorous exercise typically produces higher peak acceleration values than walking or sitting. The three-axis acceleration information obtained is shown in Fig. 3. The characteristics of the collected acceleration signal are different, which is crucial for signal recognition in the next stage.

In data collection, we first had the volunteers who received the test stand still and conducted a familiarity test after wearing it according to their movement mode. After confirming that wearing the bracelet did not affect their movement, we conducted an experimental test. The experimental subjects performed the specified sequence of activities in the designated area with videos according to the requirements, thus completing the data collection.

Motion recognition results

To more accurately explain the recognition results based on the data-driven model in this article, the recognition is evaluated by precisopm. Recall and F1 score, which can be represented from Eqs. (13)–(15): (13)${\displaystyle P=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$ (14)${\displaystyle R=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$ (15)${\displaystyle \mathrm{F1}=\frac{2\times P\times R}{P+R}.}$

TP represents true positive; FP indicates false positive; FN is the false negative. For this reason, the evaluation results are assessed by these three indicators.

This study recognizes human actions through the extracted joint angle information and hand posture acceleration and motion intensity information. The recognition results are shown in Table 2 and Fig. 4.

The proficiency of the approach in this manuscript is readily discernible from Table 1, demonstrating exemplary performance across accuracy, recall, and various other facets. Simultaneously, a sense of equilibrium pervades the recognition rates of individual data points, with each category attaining commendable recognition levels.

For a more comprehensive elucidation of the experimental findings, this study employs models predicated on diverse experimental subjects during the training phase, with the corresponding outcomes depicted in Fig. 5.

The model also has good recognition results in the recognition of specific objects. Among them, subject 2 has the highest accuracy rate of recognition results because there are no additional small actions in the movement process. It faces the camera in the process, resulting in a more accurate extraction of joint angles. In contrast, subject 1 has a more general recognition result because compared with the other two objects, there are significant differences between each group of actions during the experiment. For example, the speed of arm lifting is sometimes fast or slow, leading to low recognition accuracy, which needs to be focused on in future experiments. However, no matter the overall or single-person recognition rate, the model shows excellent recognition results with more than 95% accuracy.

The application in the art interaction

Simultaneously, to enhance the elucidation of the proposed model’s practical utility in art, this study employs the model to assess the action sequences inherent to multimedia technology interaction. Field trials were conducted with a local video special effects production company, aligning with their specific requirements. The company successfully executed action recognition within a video clip, showcasing the results in Fig. 6.

This study selected three objects for actual testing. Due to the small amount of data in the actual testing, we adopted the experimental method of leaving one and tested the three indicators mentioned above. Figure 6 shows the RNN, BP neural network, KNN, CNN and SVM models from left to right. The RNN model selected in this study is superior to the traditional BP neural network model in terms of accuracy or recall. At the same time, it can be found the performance of KNN is poor because of such relatively continuous action recognition. Through actual comparison, it is found that the identification efficiency of this article is very high and has been praised by local companies.