Physical human locomotion prediction using manifold regularization

Introduction

Predicting the human activities performed can also be referred to as human dynamics analysis and prediction (Xiao et al., 2012). The term locomotion detection is relatively novel as compared to motion detection. It is used in different capacities including locomotion prediction for estimating walking targets and redirected walking (Zank & Kunz, 2015, 2017), locomotion prediction in virtual reality for tracking space while walking (Stein, Bremer & Lappe, 2022; Bremer, Stein & Lappe, 2021), and locomotion recognition via human actions recognized (Yan et al., 2018; Ghadi et al., 2022). In the proposed system, our goal is to predict the human locomotion via multiple physical activities detection.

Some researchers proposed human locomotion analysis and prediction using motion sensors (Jalal, Quaid & Hasan, 2018; Haneche, Ouahabi & Boudraa, 2019), while others preferred vision-based sensors (Jalal et al., 2021; Gochoo et al., 2021). The applications of human locomotion prediction systems include smart home environments (Jalal et al., 2017), security and healthcare systems (Jalal, Kim & Kim, 2014), behavior mining (Jalal & Mahmood, 2019), life logging systems (Jalal, Kim & Kim, 2012), and smart surveillance (Pervaiz, Jalal & Kim, 2021). For such real-world applications (Quaid & Jalal, 2020) a variety of sensors are utilized and, the methods are still lacking in human motion prediction with minimal circumstantial information. The above-mentioned applications require recognition of both simple and complex motion patterns that is missing in literature. Due to the motion containing intervals, delays, jerks etc., it becomes difficult for the system to predict human locomotion dynamics. Therefore, considering the need, we refer simple motion patterns as static and complex motion patterns as kinematic in this research.

We propose a novel technique for the prediction of human motion (PHM) using kinematic-static patterns identification. Data is obtained from two benchmark datasets, namely, PAD (Khan et al., 2020) and GOTOV (Paraschiakos et al., 2020). For noise reduction in the signals acquired, this article develops a calibration-based filter for inertial measurement unit (IMU) data. The filter helps in reducing the errors present in the raw IMU signals. A Bessel filter is applied for the physiological data from sensory devices like electromyography (EMG) and electrocardiography (ECG). By applying these filters, we got clean data to perform further processing and achieve better results. Next, this filtered data from IMU and physiological sensors is fused together over time. Then, the data is divided into windows of 2 s each for detailed analysis of the fused signals.

Next, to decide the kinematic-static patterns among the windows (Al Shloul et al., 2022), we suggested a polynomial probability distribution. For kinematic patterned signal data, multisynchrosqueezing transform (Yu, Wang & Zhao, 2019) and hidden Markov random field (Wang, 2012) are suggested for features extraction, whereas, for static patterned signal windows, dynamic time warping (Laperre, Amaya & Lapenta, 2020) and Gaussian Markov random field (Khalid et al., 2021) are recommended. Then, two feature optimization techniques are used including quadratic discriminant analysis and orthogonal fuzzy neighborhood discriminant analysis. Lastly, manifold regularization using multiple algorithms is applied to evaluate the performance of our proposed PHM model.

The key contributions of this research are:

• Our PHM model is a novel technique of pre-extraction for kinematic and static patterned signals in order to extract and classify the complex motion patterns.

• For inertial human motion data, we designed a calibration-based filter that provides improved regulated and converged filtered data.

• For the prediction of indoor-outdoor activities, this article recommends a variety of features extraction approaches for each kinematic and static motion patterned data.

• Finally, two optimization techniques including orthogonal fuzzy neighborhood discriminant analysis along with quadratic discriminant analysis and manifold regularization algorithms provide better comparison of the proposed PHM model.

The article is divided into sections as: it provides the related work section followed by the material and methods section illustrating the overview of our proposed technique and detailed study in the noise reduction, data fusion and windowing, kinematic-static patterns decision, features extraction, features optimization, and classification using manifold regularization sub-sections. Then, the results section presents experiments conducted and outcomes of the system and finally, we concluded the article with an overall synopsis.

Related work

A variety of advanced human motion analysis approaches have been studied and utilized in indoor-outdoor physical monitoring that can be further divided into motion-based and vision-based human dynamics prediction models. Motion sensors including accelerometer (Acc), gyroscope (Gyro), magnetometer (Mag), mechanomyography (MMG), ECG, EMG, and geomagnetic (GeoMag) are used by a variety of researches. Whereas, for vision-based systems, Microsoft Kinect RGB-D cameras (Kinect), Intel Realsense (Realsense), Asus Xtion (Xtion), video cameras (video cam), and dynamic vision sensor (DVS) are utilized. Table 1 presents a literature review for human dynamics prediction via motion sensors and vision sensors based on recent studies.

From Table 1, it is observed that predicting complex human activities is still a challenge. Besides, relevant datasets need to be used with more variety of human motion. More importantly, the prediction time, complexity, and accuracy are three parameters that should be considered while recognizing human activities in PHM systems.

Materials and Methods

Data is collected from two publicly available datasets, namely, PAD and GOTOV. Figure 1 demonstrates the flow diagram of the proposed PHM model. After data collection, noise reduction techniques are applied followed by data fusion and extracting overlapping sliding windows of 2 s each. Next, the windows are used to decide for the kinematic and static patterns definition, which are further used to extract features. Then, the features are optimized using two methods named quadratic discriminant analysis (QDA) and orthogonal fuzzy neighborhood discriminant analysis (OFNDA). Finally, manifold regularization is used to verify the performance of the proposed system.

Noise reduction

Pre-processing is required on the raw data in order to reduce the noise, biasness, and other errors in the IMU and other physiological signals acquired. For this, we have proposed a calibration-based filter for IMU data and Bessel filter for other physiological data.

Calibration-based filter for IMU

A three-phased calibration-based filter is proposed for this PHM model. This filter takes care of all the noise present in the IMU signals as it has three phases for denoising the signal. It has a calibration phase, where all the three types of IMU signals including accelerometer, gyroscope, and magnetometer are calibrated and filtered for noise. Next, during the error correction phase, earth’s gravitational field is utilized to reduce errors in acceleration.

For gyroscope signals, a discrete wavelet transform technique is used to remove errors, whereas for the magnetometer signals, we used the earth’s magnetic field. Finally, in the final phase of mapping and optimization, the article proposes to map the error-corrected gyroscope signals to Quaternions. An optimal solution for the rate of change in drift is defined by gradient descent technique (Baldi, 1995). Figure 2 shows the brief description of each phase for the calibration-based filter over IMU sensor data.

Bessel filter for physiological sensors

It is a type of linear filters that provides maximum flat phase delay. Also, it keeps the shape of the wave near to original (Soni & Gupta, 2020). For physiological signals, we prefer to use a filter that will not change the characteristics of the signal itself. Therefore, we applied a Bessel filter that includes use of a transfer function as described in Eq. (1):

(1) $$\mathrm{H}\left(\mathrm{t}\right)={\displaystyle \frac{{\theta }_n\left(0\right)}{{\theta }_n\left({\displaystyle \frac{t}{{\omega }_0}}\right)}}$$where ${\theta }_n\left(t\right)$ is a reverse Bessel polynomial and ${\omega }_0$ is the cut-off frequency.

Kinematic-static patterns decision

The unique method to decide about kinematic or static patterns has been introduced in order to facilitate the PHM model for prediction.

Polynomial probability distribution

Probability density function (PDF) gives the likelihood of an outcome in a given sample space and provides an acceptable difference between both kinematic and static patterns. A goodness of fit test is defined for the PDF approximation followed by polynomial degree selection and a real valued interval is defined as $\left[x,y\right]$ (Munkhammar, Mattsson & Ryden, 2017). Equation (2) shows the PDF defined as $M^{th}$ order polynomial $P$ on the interval $\left[x,y\right]$:

(2) $$P\left(M,a\right)=w_0+w_{1}a+w_2{a}^2+\dots +w_M{a}^M$$where $w_m$ is the unknown weight for each $m\in [0,\dots ,M]$. Figure 3 represents polynomial density function comparison for kinematic and static patterns.

Features extraction

After the patterns decision for kinematic and static, the PHM model suggest to extract features for both pattern types separately. Due to the kinematic patterns being more complex and variable in nature, dynamic time warping (DTW) and Gaussian Markov random field (GMRF) are selected for the features extraction from kinematic signals. Whereas, multisynchrosqueezing transform (MSST) and hidden Markov random field (HMRF) are applied over the static signals for features extraction as they tend to have pause and delays in them.

Dynamic time warping for kinematic patterns

For kinematic features extraction, DTW is used as one of the techniques. This technique is not limited in its application and has been used in multiple areas like speech (Amin & Mahmood, 2008), biology (Petitjean et al., 2014), economics (Franses & Wiemann, 2020) etc. It is used for the time-based comparison between two signals even if the signals are stretched or shifted in time (Laperre, Amaya & Lapenta, 2020). Therefore, DTW provides good results for IMU signals. If we take two time windows $P$ and $R$, then we can represent them as shown in Eqs. (3) and (4):

(3) $$P=\left[p_1,p_2,\dots ,p_i,\dots ,p_m\right]$$

(4) $$R=\left[r_1,r_2,\dots ,r_j,\dots ,r_n\right]$$

DTW firstly calculates the distance between $P$ and $R$ using Euclidean distance formula as in Eq. (5). Then, it searches for the warping path $PT$ as shown in Eq. (6):

(5) $$d\left(p,r\right)=\sqrt{{\left(p-r\right)}^2}$$

(6) $$PT=[p{t}_1,p{t}_2,\dots ,p{t}_k]\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}max\left(m,n\right)\le k<m+n-1$$where each $p{t}_k$ represents the grid made using $p_m$ and $r_n$. Equation (7) shows the formula to calculate DTW as the cost function. Figure 4 displays the DTW calculated for two types of kinematic patterned window signals including motion and physiological.

(7) $$DTW\left(P,R\right)=min\sqrt{{\sum }_{i=1}^{k}p{t}_k}$$

Gaussian Markov random field for kinematic patterns

Random fields is the general form of stochastic process where we don’t need the real values and it can take multi-dimensional matrix or points Khalid et al. (2021). Hence, we applied it over multi-dimensional signals like motion and physiological signals together. A stochastic process becomes Gaussian when all its distributions are Gaussian normalized. To determine a Gaussian process, we need to determine its expectation function as Eq. (8) and covariance function as Eq. (9) using $s$ samples and $t$ times (Kreose & Botev, 2015). Figure 5 presents the results of two complex kinematic motion patterns using Gaussian Markov random field features.

(8) $${\tilde{\mu }}_t=\mathit{E}{\tilde{X}}_t$$

(9) $$\widetilde{\sum s,t}=\mathit{c}\mathit{o}\mathit{v}\left({\tilde{X}}_s,{\tilde{X}}_t\right)$$

Multisynchrosqueezing transform for static patterns

Synchrosqueezing transform (SST) assumes the signal under consideration to be weakly time varying and with time the SST representation becomes blurry. Therefore, an SST operation is required to be executed on already acquired SST results (Yu, Wang & Zhao, 2019). As the static patterned signals can be weak and become blurry over time, so we utilized the multiple iterative SST operations for the static signals resulting in Eq. (10) as MSST:

(10) $$T{s}^{\left[M\right]}\left(t,\gamma \right)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}T{s}^{M-1}\left(t,\gamma \right)\delta \left(\gamma -\hat{\omega }\left(t,\omega \right)\right)d\omega$$where $M$ is the iteration number $\le$2 and $T{s}^{\left[M\right]}\left(t,\gamma \right)$ represents the spread time-frequency coefficient. Figure 6 illustrates the MSST features extracted for random static windows.

Hidden markov random field for static patterns

The hidden Markov random field (HMRF) is applied (Wang, 2012), where the joint likelihood probability is described in Eq. (11).

(11) $$P\left(y|x,\mathrm{\Theta }\right)=\prod _{i}P(y_i|x_i,{\theta }_{x_i})$$where $P(y_i|x_i,{\theta }_{x_i})$ denotes the Gaussian distribution and $\mathrm{\Theta }$ shows the parameter set. To estimate the labels, MAP estimation is used by applying $\mathrm{\Theta }$ parameter set. The prior energy function is used for MAP estimation as given in Eq. (12):

(12) $$U\left(x\right)={\sum }_{cϵCl}{F}_c\left(x\right)$$where $F_c\left(x\right)$ is the potential clique and $Cl$ is the set of all possible cliques. When we extract the prior energy from potential clique, it gives us features that have similar characteristics as the clique, therefore we applied it over the static patterned signals. Figure 7 shows the results of prior energy function applied to each EM iteration, where red represents typing and blue shows resting motion patterns.

Features optimization

Features optimization using quadratic discriminant analysis

The features extracted for both kinematic and static patterned signals are non-linear. Therefore, applying quadratic discriminant analysis (QDA) is suggested instead of linear techniques. QDA is used when it is not possible to assume the activity dispersion (Bose et al., 2015). So, we calculate covariance matrix ${\mu }_m$ for each activity $m\in \left\{1,\dots ,M\right\}$. QDA can be calculated as mentioned in Eq. (13):

(13) $${\delta }_m\left(x\right)=-{\displaystyle \frac{1}{2}\log \left|\sum m\right|-{\displaystyle \frac{1}{2}(x-{\mu }_m{)}^T{\sum }_m^{-1}\left(x-{\mu }_m\right)+log{\pi }_m}}$$where ${\pi }_m$ represents $M$ activity priors and $x$ is the extracted features vector. Figure 8 shows the results of QDA applied on extracted features for both kinematic and static activities.

Feature optimization using orthogonal fuzzy neighborhood discriminant analysis

To maximize the distance between center of different classes and minimize the distance within the classes (Phukpattaranont et al., 2018), OFNDA is used as PHM model’s second features optimization technique. OFNDA also takes care of the contribution of the samples to the different activities by providing an orthogonal projection matrix (Khushaba, Al-Ani & Al-Jumaily, 2010). Equation (14) shows the proposed regularized objective function to determine the fuzzy partition matrix $F$ for grouping a collection of $d$ samples into $a$ activities:

(14) $$F_p\left(F,v\right)={\sum }_{k=1}^d{\sum }_{i=1}^a{\mu }_{ik}^p\mathrm{e}\mathrm{x}\mathrm{p}({\displaystyle \frac{\left|x_k-v_i\right|}{{\eta }_i/3}{)}^2}-\lambda {\sum }_{i=1}^a{\sum }_{k=1}^d\left({\mu }_{ik}-1\right)$$where ${\mu }_{ik}$ shows the membership grade of $k^{th}$ sample in the $i^{th}$ activity, $\lambda$ gives the language multiplier, $p$ is the fuzzification parameter, the means of input is $v_i$ from activity $i$, and ${\eta }_i$ represents the chosen radius for each activity as $max\left|x_k-v_i\right|$ having $k=1,2,\dots ,d$. Figure 9 demonstrates the OFNDA based selected features subset for PAD dataset.

Datasets description

The PAD dataset has been collected via surface electromyography (sEMG) and IMU sensors from 40 participants with equal gender distribution. It has been created to monitor the human muscles activity during routine activities including resting, typing, push up exercise, and lifting heavy objects. Total time for collection of each participant data is 70 s.

The GOTOV dataset is based on the healthcare monitoring issues for 35 elderly participants using multiple sensors, namely, accelerometer, human’s physical information sensors like heart rate. It consists of 3,400 s data for each participant. The dataset consists of sixteen activities including jumping, standing, step, lying down left, lying down right, sitting sofa, sitting couch, sitting chair, walking stairs up, washing dishes, stacking shelves, vacuum cleaning, walking slow, walking normal, walking fast, and cycling. If we take a closer look at the activities, we can see that both the datasets have multiple indoor-outdoor activities that will help in making a robustly performing PHM model.

Results

Experiment I: Manifold regularization over datasets

To evaluate the performance of our proposed PHM model, we proposed the manifold regularization using RLS, LapRLS, and Nyström LapRLS algorithms. Figure 10 represents the results in the shape of the confusion matrix over PAD dataset providing the mean accuracy as 82.50%. Accuracy is one of the vital implications of our system because it shows how accurately the proposed method was able to detect the human motion patterns. Figure 11 represents the outcomes using confusion matrix over GOTOV dataset achieving the mean accuracy as 81.90%, where the accuracies are calculated using Eq. (17):

(17) $$\mathrm{A}CC={\displaystyle \frac{correctlyclassifiedactivities}{allactivities}}$$

HA = Human actions, JP = Jumping, SD = Standing, ST = Step, LDL = Lying down left, LDR = Lying down right, SS = Sitting sofa, SC = Sitting couch, SCH = Sitting chair, WSU = Walking stairs up, WD = Washing dishes, SSH = Stacking shelves, VC = Vacuum cleaning, WS = Walking slow, WN = Walking normal, WF = Walking fast, CY = Cycling.

Experiment II: RMSE via QDA features

For optimized features via QDA, we processed for the three manifold regularization algorithms and results are described in the form of root mean squared error (RMSE) when applied over different partitions of labeled and unlabeled data for semi-supervised learning as represented in Fig. 12. RMSE points out the standard deviation of prediction errors, therefore, it is observed that RLS has provided better results when influenced by the sample proportions over QDA features for both selected datasets. So, as the sample percentage increases, the RMSE decreases.

Experiment III: RMSE via OFNDA features

All three algorithms of manifold regularization are applied over OFNDA features and we get results in the form of RMSE when applied different partitions of labeled and unlabeled data for semi-supervised learning as shown in the Fig. 13. Here, RLS delivers best outcomes when proportion of samples is low over OFNDA features for PAD and GOTOV datasets. So, when sample percentage decreases, RMSE also decreases.

Experiment IV: Comprehensive analysis over selected datasets

The experiment is performed over both PAD and GOTOV datasets to measure the results in the form of precision, recall, and F1-scores. Equations (18)–(20) represent the formulas for calculating these parameters. Table 2 shows the outcomes for PAD dataset, whereas Table 3 displays the results over the GOTOV dataset. The bold entries in both tables represents the mean values of precision, recall, and F1-score.

(18) $$Precision={\displaystyle \frac{truepositives}{truepositives+falsepositives}}$$

(19) $$Recall={\displaystyle \frac{truepositives}{truepositives+falsenegatives}}$$

(20) $$F1-Score={\displaystyle \frac{2\ast Precision\ast Recall}{Precision+Recall}}$$

Precision over both datasets explain that our proposed PHM model is good at predicting the motion patterns. Similarly, recall suggests how many times our PHM model was correctly able to predict human actions. F1-score is extracted by combining both precision and recall performance metrics, therefore it describes the properties of both.

JP = Jumping, SD = Standing, ST = Step, LDL = Lying down left, LDR = Lying down right, SS = Sitting sofa, SC = Sitting couch, SCH = Sitting chair, WSU= Walking stairs up, WD = Washing dishes, SSH = Stacking shelves, VC = Vacuum cleaning, WS = Walking slow, WN = Walking normal, WF = Walking fast, CY = Cycling.

Experiment V: Comparison with other state-of-the-art PHM models

In Wang et al. (2018), transfer learning is utilized to save the cost and computational time. They proposed a stratified transfer learning framework to learn cross-domain activity recognition. First, majority voting is used to obtain pseudo labels, then it transfers both domains into the same. Average accuracy achieved for activity recognition is 61.37%. Xia et al. (2021) proposed a novel deep learning model for human motion prediction. They have contributed by proposing a big synthetic dataset using IMU and multi-level domain adaptive learning. However, the system was able to achieve a mean accuracy rate of 73.6%. Lawal & Bano (2019) proposed an automated monitoring system for elderly health issues. They have utilized waist mounted sensors and CNN to train for the motion signals. For both static and dynamic activities, the system was able to achieve 78.0% accuracy.

Al-Naser et al. (2018) proposed a hierarchical framework that learns and classifies unidentified activities. Object recognition module, myo-armband, and activity recognition have been utilized to perform the complex activity detection and achieved 77.0% precision along with 82.0% recall rates. Kwon, Abowd & Plotz (2021) presented a body-worn model based on virtual and real IMUs. Essential information from the IMUs is extracted and presented a maximum of 80.2% accuracy rate. Paraschiakos et al. (2020) introduced an algorithm called LARA for tracking elderly motion using combinations of wearable sensors. They have considered the granularity of each action using prior biological knowledge. In two-body locations, they could achieve a maximum of 81.0% accuracy for chest-wrist-equivital sensors combination.

Kwon et al. (2021) have used the IMUTube concept for human action recognition method. They analyzed free-weight gym exercises having a range of artifacts like video noise, non-human poses, body parts occlusions etc. The IMUTube system was able to achieve a maximum of 81.5% recognition accuracy. Table 4 presents the mean accuracy comparison of PHM model with other state-of-the-art systems that shows the PHM model’s efficiency and effectiveness in bold over the other methods.

Conclusion and future work

The proposed PHM model is based on noise reduction, data windowing, patterns recognition, features extraction, and optimization techniques for feature dimensionality reduction. To evalaute the performance of the proposed model, two benchmarked datasets, namely, PAD and GOTOV, were used. For noise reduction, a state-of-the-art filter was proposed for IMU signals and a Bessel filter was utilized for the physiological sensory data. For-pre-classification, patterns were decided on the basis of polynomial probability distribution. Kinematic and static motion patterns were separated and fed to the features extraction techniques. Two optimization procedures were suggested for PHM model including QDA and OFNDA. Accuracy rates of 82.50% and 81.90% have been achieved for PAD and GOTOV datasets, respectively. PHM model has shown a great improvement when compared to the existing systems in literaure.

The drawback is that motion complexity of different patterns causes restricted decision-making for patterns identification. The selected algorithms were not able to detect extremely complex patterns. Another limitation is the size of datasets used. In future, this work can be applied to larger datasets with diverse range of activities in order to predict additional motion patterns.

Supplemental Information