Pattern transition recognition based on transfer learning for exoskeleton across different terrains

Experimental protocol

This study involved human participants. All ethical and experimental procedures were approved by the ethical committee for sports science experiments in Beijing Sport University (Application No. 2019007H) and conducted in accordance with the protocols for human motion experiment.

Fifteen healthy participants (5F/10 M, Height: 173.2 $\pm$ 1.7 cm, Weight: 69.8 $\pm$ 4.8 kg, Age: 28.2 $\pm$ 1.3 years) are voluntary and supplied written informed consent to take part in the study. As shown in Figs. 1A–1E, the participants with exoskeleton engage in walking activities in scene. All the participants were divided into three groups. They walked on three terrains of level ground, stair, and ramp, severally. Each group held up multilevel loads (0, 20, 40 kg) on the back of human body respectively. M0, M20, and M40 denote multilevel loads of 0, 20, and 40 kg, severally. First group went straight ahead along the concrete road. Second group climbed up the stairs and then went back the same way. Third group went up the ramp with a slope gradient and then followed it back to level ground. Finally, three groups got through the walking process described above.

The equipment configuration

The STM32F765II microcontroller applies the advanced ARM Cortex-M7 32-bit to the development board in the system. Data transmission goes through RS485 bus and operates at the speed of 2 Mb/s with the sampling frequency of 100 Hz. PC equipped with AMD Ryzen 7 6800H CPU, 32 GB RAM, and Windows 11 (64-bit) serves as the data processing unit. It is Anaconda 3, Python 3.10, and PyTorch 2.0 in the software environment. A random algorithm is executed to enable real-time performance verification using multithreading.

Data preprocessing

Each participant wears the lower limb exoskeleton. FSRs and IMUs attached to feet, shanks, thighs and trunk of both mechanical legs in exoskeleton are applied to acquire spatial and temporal data. The body trunk along vertical direction and the thigh are computed to gain hip angle. The thigh and the shank are calculated to gain knee angle. The foot IMU in horizontal direction serves as the foot angle. Clinical gait analysis of human body is applied to define joint angles.

It is indispensable to preprocess original data to promote multidimensional features consistent and achieve optimal performance. The original data is calculated into ground reaction force and joint angle, which apparently expresses the diversity in gait analysis. Degree is used as the unit for measuring joint angle. The calculated joint angles are used as inputs to the dataset. The packet loss takes advantage of linear interpolation to compensate the loss in data delivery. Furthermore, joint angle data conducted by normalization plays as the input data for model training. Therefore, it is significant to preprocess chain in the dataset.

Evaluation indicator

Comprehensive evaluation index system composed of Pre-T, Pre-T/GC, confusion matrix, accuracy, precision, recall, and F1 is used to assess performance. Each of these indexes is interpreted as shown below.

Prediction time (Pre-T) is defined as the interval between a vital moment and the point of heel strike. A vital moment represents predicted point in the machine learning model. Pre-T/GC represents the proportion of a gait cycle duration consumed by prediction time.

(1) $$\mathrm{P}\mathrm{r}\mathrm{e}-\mathrm{T}/GC=\frac{t}{T}$$where t denotes prediction time, T signifies a gait cycle.

The confusion matrix is generally used to evaluate classification performance. It consists of four key components and summarizes the model prediction by comparing them with true labels. True positive (TP) indicates that positive samples are correctly predicted as positive. True negative (TN) denotes that negative samples are correctly predicted as negative. False positive (FP) refers to negative samples that are incorrectly predicted as positive. False negative (FN) refers to positive samples that are incorrectly predicted as negative.

Accuracy represents the ratio of correctly classified samples to the total number of samples.

(2) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{TP+TN}{TP+TN+FP+FN}.$$

Precision is the ratio of true positive predictions to the total number of samples predicted as positive.

(3) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{TP}{TP+FP}.$$

Recall is the ratio of true positive predictions to the total number of actual positive samples.

(4) $$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{TP}{TP+FN}.$$

F1-score combines precision and recall into a single metric by calculating their harmonic mean.

(5) $$\frac{2}{F_1}=\frac{1}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}+\frac{1}{\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}.$$

It is essential to acquire the assessment criteria to generate differences in comparative approaches. Every experiment is repeatedly conducted to verify the statistical rule of the network model.

Situation of pattern transition

There are multiple locomotion modes transition across diverse terrains, such as level ground, stair, and ramp. It is composed of standing phase and swing phase in a gait cycle depicted in Fig. 2. Heel strike not only acts as the end point of last gait cycle, but also plays as the starting point of subsequent gait cycle. Heel off regards as the dividing line of standing phase and swing phase. Pre-T is defined as the interval between a vital moment and the point of heel strike. Pattern transition appears a critical interval time before subsequent locomotion mode. In addition, a series of cell array are located in preceding transition to subsequent locomotion mode. The length of cell array is 60 ms. As described in Fig. 3, Pattern transition is explicit between two locomotion modes adjacently, such as LW $\to$SA, LW $\to$SD, LW $\to$RA, LW $\to$RD, SA $\to$LW, SD $\to$LW, RA $\to$LW, RD $\to$LW.

Dataset architecture

All the datasets consist of three types of sub-datasets in three levels of weight, which come from human walking on different terrains. Pattern transition occurs in swing phase of the gait cycle in the curves of hip angle, knee angle, foot angle, and foot pressure depicted in Figs. 4A–4H. Pattern transition is explicit between two locomotion modes adjacently, such as LW $\to$SA, LW $\to$SD, LW $\to$RA, LW $\to$RD, SA $\to$LW, SD $\to$LW, RA $\to$LW, RD $\to$LW. It is eight patterns transition among five dynamic locomotion modes in sequence, such as LW, SA, SD, RA, and RD. The proportion of the training set and the testing set in a dataset is 4:1. The input features are composed of left and right hip angle, left and right knee angle, and left and right foot angle. The multiple features and the length of the unit sample are set to 6 and 6 in input layer, respectively. The matrix size of a cell array consisting of the unit sample data is set to 6 × 6.

Transfer learning

Transfer learning attracts widespread attention in the rapid development of artificial intelligence. It is designed to transfer knowledge and adapts the model from source domain to target domain. It contributes to reduce the sample size requirement of target domain and accelerate learning process, which improves generalization performance of the model. There are primarily transfer learning methods, such as model transfer, feature transfer, and relationship transfer. As shown in Fig. 5, model transfer refers to migrating the model from source domain to target domain and fine-tuning hyperparameters to adapt to classification task in the target domain.

The model transfer method involves multiple steps, such as choosing a source model, determining a transfer strategy, adjusting model hyperparameters, and evaluating model performance. It is necessary to select a suitable pre-trained model to meet the demand of the target task and the characteristics of source domain. An appropriate transfer strategy depends on the higher similarity between the source domain and target domain. It fine-tunes the model hyperparameters on the target domain, which usually involves in adjusting the parameters of the preceding layers and the final layer to satisfy the needs of the target assignment. The model performance on the target domain is estimated by evaluation metrics. As an efficient machine learning method, transfer learning has advantage on addressing challenges of data scarcity and domain differences.

TCN

As shown in Fig. 6, temporal convolutional network (TCN) is a model framework of deep learning developed for processing serial data. It combines the parallel processing capability of CNN with the long-term dependency of recurrent neural network (RNN) and makes it a powerful tool for sequence modeling tasks. It presents the core characteristics of sequence modeling and then defines the network structure.

An input sequence $x_0,x_1,\dots ,x_T$ and the corresponding output $y_0,y_1,\dots ,y_T$ are at each time step. When predicting the output $y_t$ is at a specific time step $t$, the input $x_0,x_1,\dots ,x_t$ are used to observed. A sequence modeling network is an arbitrary function $f:X^{T+1}\to Y^{T+1}$ that generates mapping as follows.

(6) $${\hat{y}}_0,{\hat{y}}_1,\dots ,{\hat{y}}_T=f(x_0,x_1,\dots ,x_t).$$

To satisfy the causality constraint, $y_t$ only depends on $x_0,x_1,\dots ,x_t$.

The learning objective in sequence modeling is used to search the network that minimizes the expected loss $L(y_0,y_1,\dots ,y_T,f(x_0,x_1,\dots ,x_T))$ between the actual outputs and the predicted values, where the sequences and outputs are drawn from the certain probability distribution.

Causal convolution. It means that the output of TCN model depends on the current and past points. In standard convolution operations, each output value is based on its surrounding input values, such as current points and past points.

However, the weights in causal convolution are applied to the current and past input values and ensures the directionality of the information flow and preventing future information from leaking into the current output. Zeros are typically padded on the right side of the convolution kernel so that the current and past data is used to compute the output.

(7) $$y(t)=\sum _{i=0}^{k-1}f(i)\cdot x(t\text{-}i)$$where $k$ refers to the kernel size, $x$ denotes the input series, $f$ represents the convolution kernel.

Dilated convolution. It is employed by TCN to expend the receptive field. The gaps of dilated convolution are inserted between kernel elements and allows for broader context capture. Therefore, it is able to grasp the contextual information in the input data. The dilation factor determines the spacing between elements in the convolution kernel. The mathematical representation of dilated convolution as follows.

(8) $$y(t)=\sum _{i=0}^{k-1}f(i)\cdot x(t\text{-}d\cdot i)$$where $d$ is the dilation rate.

Residual connection. TCN employs residual connections to alleviate the vanishing gradient problem and improve the training efficiency of deeper networks and optimize model performance. Residual connections are a key component of residual network (ResNet). In residual connections, the output of a certain layer in the network directly added a few layers to another layer, which forms a skip connection.

Given an input $x$, the output is $F(x)$ after going through several layers. The final output is $x+F(x)$. It allows gradients to flow directly back to earlier layers during backpropagation. It reduces the issue of vanishing gradients and enables effective training of deeper architectures. The output of a residual block is expressed.

(9) $$\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{p}\mathrm{u}\mathrm{t}=\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(\mathrm{i}\mathrm{n}\mathrm{p}\mathrm{u}\mathrm{t}+F(\mathrm{i}\mathrm{n}\mathrm{p}\mathrm{u}\mathrm{t}))$$where F represents the combination of the convolution layer and the activation function.

The basic structure of TCN consists of multiple residual blocks. Each residual block includes dilated causal convolution layer, layer normalization, ReLU activation function, and dropout layer.

Attention

Attention mechanism allows the model to dynamically adjust the attention weights in processing data depicted in Fig. 7. It highlights important data and ignores irrelevant details. It imitates human brain to selectively focus on concrete details selectively, which demonstrates exceptional performance in processing sequential and high-dimensional data.

The input to attention mechanism typically is composed of three parts. Query (Q) represents the processed data currently in the current time step. Key (K) denotes the features of the reference data used to match query. Value (V) represents the content associated with each key and contributes to the output of the attention mechanism.

The attention mechanism calculates the relevance between the query and the key and then performs the weighted average of the values. The formula is mentioned below.

(10) $$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q,K,V)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$$where $Q{K}^{\mathrm{\top }}$ calculates the dot product similarity between the query and the key, the scaling factor $\sqrt{d_k}$ prevents gradient instability from excessive dot product values. Softmax converts the similarity into weights as probability distribution.

Spatial attention

Spatial attention regards as the adaptive mechanism of spatial region selection to enhance model performance and efficiency in deep learning. Generally, spatial attention mechanism achieves enhanced representation of effective spatial features and suppression of irrelevant spatial features by gaining attention weights for each spatial position.

As shown in Fig. 8, spatial attention mechanism is conducted by the following steps. Assume that the feature map has a size of $CHW$, C is the number of channels. H and W are the height and width, respectively.

Feature mapping. Each spatial position in the feature map is mapped to an attention score space. A small convolutional kernel implements the process. It is aimed at generating a corresponding attention weight for each spatial position.

Weight calculation. Attention weights for each position are computed and then normalized into a probability distribution by softmax. The sum of attention weights across all positions equals 1, which prompts the model to emphasize the significant areas and disregard less relevant ones.

Weighted feature fusion. Those weights are utilized to the original feature map that results in the weighted feature map. It enhances the feature representation of regions with high attention weights and suppresses those with low attention weights.

Framework of transfer learning model based on TCN-SA

As shown in Fig. 9, it is the framework of transfer learning model based on TCN-SA. Transfer learning refers to migrate TCN-SA model from dynamic pattern detection to classification task and fine-tunes hyperparameters to adapt to pattern transition recognition.

The hyperparameters comparison

In terms of many locomotion modes transition across different terrains, the hyperparameters comparison experiments are mainly conductive to analyze the influence of channel numbers, kernel size, and learning rate on accuracy of pattern transition recognition shown in Fig. 10. Multigroup experiments in transfer learning model based on TCN-SA are conducted as follows.

Part 1: In the hybrid model, kernel size 6 and learning rate 0.005 are set as the constants. It performs the comparative experiments of the channel numbers. It demonstrates the relationship of channel numbers and accuracy in the proposed model. To verify the impact of channel numbers on accuracy, they are set to 24, 32, 48, 64, and 72 respectively in Fig. 10A. When the channel number is 48, the mean value reaches peak.

Part 2: In the hybrid model, the channel number 48 and learning rate 0.005 play as the constants. It shows the comparative experiments of kernel size. To confirm the impact of kernel size on accuracy of the hybrid model, they are set to 4, 6, 8, 10, and 12, severally in Fig. 10B. When the kernel size is 6, the mean value achieves a high level.

Part 3: In the hybrid model, the channel number 48 and kernel size 6 serve as the constants. Learning rate is applied to the comparative experiments. It displays the correlation of learning rate and accuracy in the proposed model. To verify the impact of learning rate on accuracy in the proposed model, they are set to 0.001, 0.005, 0.01, 0.015, and 0.02 respectively in Fig. 10C. When learning rate is 0.005, the mean value achieves the higher level.

Performance of transfer learning model based on TCN-SA

In Table 1, the framework of the transfer learning model is evident in the aspect of the hyperparameters choice. The kernel size and the number of channels in the convolutional layer are 6 and 48 separately. In model training, the batch size and the epochs are set to 6 and 100, severally. The learning rate and the scale factor are 0.005 and 0.2, respectively.

As described in Figs. 11A, 11B, the performance of TCN-SA takes advantage of the layered K-fold cross validation. The K values are set to 3 and 5, severally. Each fold’s score is in the range of 97% to 99%. In the training process, the dataset is divided into K groups with same size. K−1 folds serve as training and the Kth fold remains validation. In cross validation, the stability of the evaluation result largely depends on the K value. The layered K-fold confirms that the sample proportion for each locomotion mode transition is steady in every fold.

It takes the calculation time in resource consumption to model training and testing once. As shown in Fig. 11C, the calculation time for multigrade loads is approximately 0.51 ms, 0.35 ms, and 0.32 ms, respectively. It reduces computational complexity and saves resource cost. To verify the prominent performance of TCN-SA on pattern transition recognition, the following experiments are carried out in time analysis.

Pre-T analysis in pattern transition

Prediction time (Pre-T) is analyzed to verify the efficiency of the proposed model in the real-time process of pattern transition. The recognition effects of Pre-T in multilevel loads are depicted in Fig. 12. During eight locomotion modes transition, the hybrid model can display the real-time performance of detecting pattern transition between two neighboring states steadily.

Pre-T and the errors in M0 serve as the referenced time. In M20, Pre-T of LW $\to$RD, RA $\to$LW, LW $\to$RA, SD $\to$LW, SA $\to$LW, and LW $\to$SA declines rapidly. Pre-T of RD $\to$LW and LW $\to$SD grow gradually. The errors of LW $\to$SD, SA $\to$LW, and LW $\to$SA rise rapidly. The errors of RD $\to$LW, LW $\to$RD, and RA $\to$LW drop off dramatically.

During pattern transition recognition, Pre-T of next locomotion mode in three levels of weight are 240–600 ms, 200–410 ms, and 120–420 ms before the step into that locomotion mode. On the whole, the mean prediction time in M20 is more stable than that in M0.

In M40, Pre-T of LW $\to$RD, RA $\to$LW, LW $\to$RA, SD $\to$LW, LW $\to$SD, SA $\to$LW and LW $\to$SA go down dramatically. Pre-T of RD $\to$LW rises distinctly. The errors of LW $\to$RD, RA $\to$LW, LW $\to$RA, SD $\to$LW, and LW $\to$SD drop off slowly. The error of LW $\to$SA rises rapidly.

The average prediction time decreases distinctly in multigrade loads. The errors in three levels of weight are 65–220 ms, 45–170 ms, and 36–130 ms before Pre-T of next locomotion mode. In the proposed model, most locomotion modes transition between two adjacent states are steadily predicted before next locomotion mode.

Pre-T/GC analysis in pattern transition

Pre-T/GC is shortened form of the ratio of prediction time to a gait cycle. Pre-T/GC is statistical to confirm the real-time performance of the proposed model during pattern transition. It is essential for identifying pattern transition between two states adjacently. Most transitions begin early in swing phase prior to next locomotion mode.

As described in Fig. 13, it emphasizes Pre-T/GC and the errors during eight locomotion modes transition in multigrade loads. In M0, their proportion of Pre-T/GC are more than 25% in LW $\to$SA, SD $\to$LW, and LW $\to$RD, respectively. In M20, their ratio of Pre-T/GC are more than 25% in RA $\to$LW and RD $\to$LW. In M40, their ratio of Pre-T/GC are more than 25% in LW $\to$SA, SD $\to$LW, LW $\to$RD, and RD $\to$LW. For locomotion modes transition, Pre-T/GC in multilevel loads are 14.2–36%, 14.82–28.22%, and 7.4–29.1% in a gait cycle. The errors of Pre-T/GC in multigrade loads are 3.68–14.97%, 2.85–8.8%, and 2.14–9.65%, respectively. The proposed model can seamlessly detect transition between two states adjacently, which presents the real-time effect in triple physical loads.

Performance assessment

In this part, the transfer learning method based on TCN-SA has prominent performance on pattern transition detection by contrast with TCN-attention, TCN, ResNet, and LSTM in the assessment criteria of accuracy, precision, recall, and F1. Confusion matrix displays true values and predicted values in standard of division to grasp how the accuracy continuously changes.

In TCN-SA, Figs. 14A–14C depicts that the precision, recall, and F1 of eight locomotion modes transition in M0, M20, and M40 gradually go up. As described in Figs. 14D–14F, the accuracy of their confusion matrixes in multilevel loads achieves 97.46%, 97.62%, and 98.21%, severally.

In TCN-attention, the accuracy of their confusion matrixes in multigrade loads reaches 96.38%, 96.43%, and 96.13%, severally. As shown in Figs. 15A–15C, the precision, recall, and F1 of eight locomotion modes transition in three levels of weight gradually increase and then go down.

In TCN, the accuracy of their confusion matrixes in M0, M20, and M40 achieves 95.89%, 96.83%, and 94.94%, severally. As displayed in Figs. 16A–16C, the precision, recall, and F1 of eight locomotion modes transition in triple physical loads gradually go up and then sharply decrease.

In ResNet, the accuracy of their confusion matrixes in M0, M20, and M40 reaches 96.35%, 95.12%, and 96.3%, respectively. As described in Figs. 17A–17C, the precision, recall, and F1 of eight locomotion modes transition in multilevel loads go down in difference.

In LSTM, the accuracy of their confusion matrixes in M0, M20, and M40 achieves 81.49%, 86.2%, and 88.52% severally. As illustrated in Figs. 18A–18C, the precision, recall, and F1 of eight locomotion modes transition in multigrade loads dramatically decrease.

Hence, the experimental results demonstrate that the accuracy of TCN-attention, TCN, ResNet, and LSTM is less than 97%. The transfer learning method based on TCN-SA has prominent effect in assessment criteria of precision, recall, and F1. It is apparent that the proposed method adequately improves the efficiency of pattern transition recognition.

Generalization capability

As described in Fig. 19A, the comparative experiments of one-to-one and many-to-many groups are carried out to clarify the generalization capability. In the one-to-one experiments, lower limb motion data from a single subject is used for both training and testing the hybrid model. In the many-to-many experiments, lower limb motion data from multiple subjects is used to train the hybrid model, which is then evaluated on the preprocessed data from the separated multi-subject sample. The results of multiple experimental groups demonstrate that the hybrid model provides good performance for individual independence. The proposed method shows excellent consistency in recognizing pattern transition. Moreover, the hybrid approach performs strong generalization capability across 15 subjects in the self-constructed dataset.

As shown in Fig. 19B, it demonstrates the effectiveness of pattern transition recognition across many locomotion modes on two public datasets. In this study, two public datasets (GIOT-2024 and EPIC-2023) from EPIC Lab in Georgia Institute of Technology are the human lower-limb biomechanics and exoskeleton wearable sensors datasets. There are cyclic activities (level ground, stair ascending, stair descending, ramp ascending, and ramp descending) from several subjects without loads in normal speed in three terrains. The collected data from IMU and force plate attached to exoskeleton is at the sampling frequency of 200 Hz. The diversity of indoor activities provides a valuable basis for validating the robustness and generalizability of the proposed approach.

Comparison to the approaches

In Table 2, this study is in contrast with many international advanced approaches in certain diversity background. We proposed the transfer learning method based on TCN-SA for eight locomotion modes transition in multilevel loads using IMUs on different terrains. Furthermore, the average accuracy of pattern transition recognition is 97.76%.

Figueiredo et al. (2020) studied that Gaussian support vector machine (SVM) model had the desired effect in nine locomotion modes transition recognition using IMUs. Papapicco et al. (2021) proposed that binary tree method predicted motion intention in next step using IMUs. Camargo et al. (2021) put forward the combined classifier of latent Dirichlet allocation (LDA), SVM, and deep belief network (DBN) for pattern transition detection using sEMG and IMUs. Su et al. (2019) studied CNN model for pattern transition detection using IMUs in healthy and disabled participators. Liu, Wang & Gutierrez-Farewik (2021) came up with the muscle synergy-inspired (MSI) method of non-negative matrix factorization (NMF) and non-negative least squares (NNLS) for 27 locomotion modes transition using sEMG and IMUs. During pattern transition detection, prediction time of next step was about 300–500 ms before the step into that locomotion mode. Liu & Gutierrez-Farewik (2023) demonstrated that muscle synergy patterns (MSP) method based on NMF, NNLS and LSTM for seven locomotion modes transition using sEMG and force plate reached accurate moment prediction.

Hence, it is evident that internal factors of wearable sensors significantly affect pattern transition detection, such as sensor types and corresponding application scopes. There is a clear distinction between indoor and outdoor scenes in current studies. In addition, the differences between healthy and mobility-impaired individuals reported in many studies may lead to variation in detection performance.