Artificial neural network-based ground reaction force estimation and learning for dynamic-legged robot systems

Force estimation

Recent studies have investigated force estimation in various applications. In Azimi et al. (2018), researchers proposed two observer-based methods. Sliding mode observer and adaptive observer were proposed to estimate GRF in a powered three-DoF prosthetic leg. In Yigit et al. (2020), the authors used an observer-based method, an extended Kalman filter method, and an artificial neural network (ANN) trained with the physical-world datasets to estimate external force/torque in a variable stiffness actuator. Furthermore, a novel ANN-based method was proposed by Fallahinia & Mascaro (2022), which utilized a convolutional neural network to estimate the tactile grasp force of a human finger from image data. Other force estimation methods were proposed in Liu et al. (2018), Fakoorian et al. (2016), Sakamoto et al. (2023), Eguchi et al. (2019), and Xia & Kiguchi (2021).

MLP

In machine learning, MLP is a type of artificial neural network that consists of an input layer, one or more hidden layers, and an output layer. Each layer is made up of one or more nodes, also known as neurons, which perform a linear transformation of the input followed by a nonlinear activation function. MLP is a versatile tool and can be utilized for many tasks, such as classification, regression, and feature extraction. With the increasing depth of neural networks in deep learning, MLPs are also widely used due to their flexibility and effectiveness.

In addition, MLP is an artificial neural network that can be used for capturing complex and nonlinear relationships between input and output variables (Specht, 1991; Hornik, Stinchcombe & White, 1989). With sufficient training data and computational resources, MLPs can approximate the highly nonlinear function with high precision. MLPs are made up of interconnected layers of nodes, where each node applies a nonlinear activation function to the weighted sum of its inputs. These functions allow the MLP to learn complex and nonlinear transformations of the input data. While training an MLP model, the MLP adjusts the weights and biases of its nodes to minimize a loss function that measures the difference between the model’s estimations and the ground-truth values of the output variable.

In regards to robot control, the appropriate selection of activation function, number of layers, and number of nodes are essential for fast and accurate estimation with MLP. The activation function plays a significant role in determining how the neurons in each layer transform the input data. The nonlinearity and smoothness of various activation functions can impact the accuracy and speed of MLP’s output estimation. While a highly nonlinear activation function can effectively capture complex patterns in the data, it increases the computation burden. Besides, the number of layers and nodes in the MLP architecture are essential factors that influence estimation speed and performance. A deeper network with more layers can potentially capture more complex relationships in the data, but this may come at the expense of increased computation time. Similarly, increasing the number of nodes in each layer may enhance the model’s nonlinearity, but it may also raise the risk of overfitting and slow down the estimation speed.

Overall, MLPs are a versatile and potent tool for nonlinear regression, capable of capturing complex relations in the given data and providing reliable estimations when their parameters are tuned to ensure optimal performance for a specific task.

Sim2Real

Sim2Real is an approach that involves training an artificial neural network in a simulated environment and transferring it to a physical world environment. This method aims to reduce the time and cost associated with training neural networks while maintaining optimal performance in the real world. However, one of the main challenges of Sim2Real is the domain gap between the simulation and the physical world environments (Höfer et al., 2021), which can result in decreased performance when the trained network is used in the physical world. To address this challenge, domain randomization and domain adaptation are two main approaches used to overcome the issue of domain gap.

Domain randomization is a technique that involves randomizing parameters such as textures, masses, and dynamics during the simulation. This approach exposes the model to various conditions it might encounter in the physical world, making it more robust and better equipped to handle unexpected situations and variations. The ultimate goal is to enhance the model’s credibility in the physical world by allowing it to learn to adapt to different environments through domain randomization. In Peng et al. (2018), dynamics randomization is applied to develop reinforcement learning policies that adapt to highly different dynamics to overcome modeling errors in the physical world.

Domain adaptation is used to bridge the gap between the simulation and the physical world environments by adapting the knowledge learned from the simulation to the physical world. Domain adaptation involves re-training the trained network with datasets collected in the physical world to learn the discrepancies between the two environments. In domain adaptation, feature-based methods, instance-based methods, and adversarial methods are included. In feature-based domain adaptation, the aim is to learn features that are invariant to both domains. Instance-based domain adaptation uses a few labeled examples from the target domain to modify the model to the target domain. Adversarial domain adaptation focuses on training a model to make the two domains indistinguishable, thereby enabling the model to generalize well to the physical world. In Ganin et al. (2016), the adversarial domain adaptation method was applied to train an artificial neural network to not classify the domain while maintaining high performance in both domains.

System overview

In this section, we present the overall hardware configuration, including a point-foot robot and sensors, for GRF estimation. Figure 1A shows our point-foot robot, which has two-DoF with hip/knee joints connected to a linear guide for motion stability, and the parameters and coordinate definition of the robot are respectively shown in Table 1 and Fig. 2B. In order to reduce the mass of the links, we used stiff, lightweight aluminum as the material for the thigh/calf links. Two planetary reduction geared motors were used as actuators for the two joints, and the motor specifications are shown in Table 2. The actuator for the hip joint was directly connected to the output side of the motor. In contrast, the actuator for the knee joint was connected to the motor with a timing belt, which was aligned with the hip motor to reduce inertia. Figure 2A shows the structure of the timing belt system in the robot. We set up the radius and number of teeth to be equal for the input and output pulleys so that the speed ratio was 1:1. Figure 3 shows the overall communication system in our robot for training a GRF estimation network. CAN communication was used to connect the motor driver to the control PC at a frequency of 1 kHz and a bitrate of 1 Mbps. The load cell was used to measure the physical world’s GRF. Serial communication was used to obtain measured GRF at a frequency of 1 kHz.

The timing belt and pulley mechanism reduced the inertia while locating the actuator position far from the joint position. In this regard, sufficient belt tension was required to transmit the rotational force generated by the motor using the belt. The tension works to distribute the rotational force, so it can be regarded as the static friction between the two different pulleys. Therefore, the timing belt and pulley mechanism affected the actuating model by friction and tension of the belt and is not affected by the backless of the planetary gear in the motor. This effect resulted in the difference between the simulation model and the real hardware model. This difference was regarded as the difference as a domain gap for the Sim2Real problem.

Including the specific reason mentioned above, various and un-modeled cause GRF differences between the ground truth and the estimated value obtained from the equation of the motion. Figure 1B shows data collected in real experiments which highlight these differences. It is important to focus on this phenomenon and to resolve it in the simple dynamic model for the legged system.

The GRF estimation proposed in this article was based on the neural network trained with the data collected in a simulation environment and the real world. As opposed to the ideal data collecting condition in a simulation environment in terms of high sampling frequency and none of noise, data collected in real experiments are limited by the following conditions: The maximum bandwidth of the data is limited by the specification of the sensor system collecting GRFs, including the load cell, the processor of the analog to digital converter (ADC), etc. Also, the data contains sensor noise. For these reasons, the sensors used to measure the robot’s motion and the ground contact forces may not provide enough information to accurately estimate the GRFs.

Training in the simulation

In this section, we described how we collected data from the robotics simulation (Hwangbo, Lee & Hutter, 2018) and trained the GRF estimator using the collected datasets with PyTorch C++ API (Paszke et al., 2019). We selected the MLP model as the estimation model and chose the appropriate training parameters. We also evaluated the performance of the trained GRF estimator in the simulation.

Collecting datasets

In collecting the training datasets in the simulation, it was assumed that the joint positions, velocities, and torques could be measured. The desired height of the robot from the ground ($q_3^d$) is set as sinusoidal functions as described in Eq. (1). Inspired by the Fourier series, which can express all conceivable motions as a combination of sinusoidal functions, the amplitude (h_A), frequency (f), and offset (h₀) were varied to produce a diverse range of robot motions while collecting datasets. The constraints on each parameter in the sinusoidal function are set as Eq. (2). (1)${\displaystyle q_3^d=h_{A}sin\left(2\pi ft\right)+h_0}$ ${\displaystyle 0.05m\le h_A\le 0.10m}$ (2)${\displaystyle 0.1Hz\le f\le .0Hz}$ ${\displaystyle 0.15m\le h_0\le 0.30m.}$

In Eq. (3), the analytic inverse kinematics method was used to derive the desired states of each joint ($q_1^d,q_2^d$) based on the generated desired height of the robot ($q_3^d$) while adhering to kinematic constraints.

${\displaystyle q_1^d=\text{arccos}\left(\frac{q_3^d}{l_1+l_2}\right)}$ (3)${\displaystyle q_2^d=-2q_1^d.}$

We used a joint PD controller (Eq. (4)) as a motion controller to track the desired joint states of the robot. (4)${\displaystyle \tau =K_p\left(q^d-q\right)+K_d\left({\dot{q}}^d-\dot{q}\right).}$

During the simulation, the robot’s states (Eq. (5)) and measured GRF (Eq. (6)) were collected for supervised learning with a sampling frequency of 1 kHz. A total of 1,303,309 datasets were collected. Of the collected datasets, 90% were used for training the model and the remaining 10% for the validation of the trained model. The overall framework for collecting datasets is shown in Fig. 4. (5)${\displaystyle x=\left[q_1,q_2,\dot{q_1},\dot{q_2},{\tau }_1,{\tau }_2\right]}$ (6)${\displaystyle y=\left[F_z\right].}$

Training MLP model

The MLP network model was chosen for the GRF network due to its lower computational burden compared to other models, such as the convolutional neural network (CNN). The overall hierarchy of the MLP network model for GRF estimation is illustrated in Fig. 5.

The input layer of the network was comprised of histories of joint positions and velocities, as well as current joint positions, velocities, and torques. According to Hwangbo et al. (2019) in ‘Modeling the actuation’, two histories with a time interval of 20 ms were selected to learn the network’s dynamics.

The hidden layer of the MLP network consisted of three major components: (1) the number of hidden layers (represented as N in Fig. 5); (2) the number of nodes (represented as M in each layer); and (3) the type of activation function in each layer. These components had a crucial role in the estimation time and performance of the trained network. If MLP networks are trained with large values of M and N, they can easily overfit and require significant time for estimation. Additionally, the type of activation function also affects the calculation time and performance of the trained networks. Hence, the careful selection of these three components is crucial to ensure that the trained network fit our system.

The training of the MLP network followed Algorithm 1 in Table 3. To achieve higher performance, we used the adaptive batch size algorithm based on ADABATCH (Devarakonda, Naumov & Garland, 2017), which gave a higher performance associated with large batches, maintaining the better test accuracy of small batches. For faster convergence, we chose the Adam optimizer (Kingma & Ba, 2014). The batch size was updated twice and the checkpoint was lowered when the model’s loss fell below the checkpoint. The training process stopped when the current number of epochs reached 300.

To discover an appropriate MLP model for GRF estimation, we employed supervised learning using different types of activation functions and varying numbers of hidden layers and nodes. The experiments were conducted on 18 different models, and the results are presented in Table 4. RMSE on validation datasets and single estimation time were measured for performance evaluation.

In the MLP models tested, those utilizing the swish activation function displayed lower RMSE values compared to those utilizing the softsign activation function. The estimation time for the swish activation function models was slightly longer. However, the difference in estimation time between models with the same number of layers and nodes but different activation functions ranged from only 17 to 61 µsec, which does not significantly affect the control system. Additionally, results from model 8 and model 9 indicated that utilizing the softsign activation function with a relatively large number of layers and nodes failed to produce successful learning. Thus, utilizing swish as the activation function in the hidden layer is recommended.

In order to determine the optimal number of layers for the GRF estimation network, we compared the results of model 10 to model 18. From our analysis, we discovered that the RMSE decreased as the number of layers increased up to nine, but increased when it exceeded that number. Additionally, we observed that the estimation time increased proportionally to the number of layers. Based on these findings, we selected nine layers as the optimal number for the network’s performance.

After determining the number of layers, we proceeded to select the optimal number of nodes by comparing model 13, 14, and 15. There was a trade-off between the error and estimation time as the number of nodes increased. However, even in model 15, which used 32 nodes, the estimation time was only 661 µsec, indicating that about 1,500 estimates can be conducted in a second. This was sufficient for our control system, which had a control frequency of 1 kHz.

As a result, we chose model 15 as our GRF estimation MLP network, as it utilized the swish activation function, had nine layers, and 32 nodes in each layer. This model was determined to provide the optimal balance between estimation accuracy and computational efficiency for our specific control system. (The performance of the selected model is shown in bold in Table 4).

GRF estimation in the simulation

In order to evaluate the performance of the trained GRF estimator, we conducted experiments to estimate GRF using the trained MLP network in the simulation for various robot motions. The overall process to estimate GRF in the simulation is shown in Fig. 6A. During the estimation of GRF in the simulation, the robot’s states were sampled at a rate of 1 kHz and stored in the state buffer. From the state buffer, the time-series states of the robot were fed into the input of the trained network. The trained model outputs the estimated GRF at each time period.

The trained GRF estimator was evaluated in different cubic motions that were not used in training. The results of the experiments are shown in Fig. 6B. In Fig. 6B, cases (1) to (4) are the results of GRF estimation when the robot followed the cubic polynomial trajectories to the target position in 2 s. Cases (5) to(8) are GRF estimation results when the robot moves in a relatively short time of 0.5 s. From the results, the RMSEs are less than 0.1 N in all cases, indicating that the estimation performance of the trained network was satisfactory.

The results of the experiments demonstrated that the proposed GRF estimator performed sufficiently for the different cubic motions, implying its ability to estimate GRF for non-trained motions. In addition, the trained MLP network, which was trained on motions generated using the Fourier Series-inspired strategy, has the ability to estimate GRF for a wide range of motions.

Transfer to the physical world

In this section, we present the strategy used to employ the GRF estimation network, which was trained with the simulation in the physical world scenarios. We evaluated the performance of the network when directly applied to the physical world, and we introduced a Sim2Real approach that aims to bridge the gap between the simulation and the physical world.

Naive Sim2Real transfer

We attempted to naively estimate the GRF in the physical world by employing the GRF estimator trained with the simulation datasets. The procedure to estimate GRF in the physical world has the same process as the estimation of GRF in the simulation and is depicted in Fig. 7A. The point-foot robot was controlled in a 1 kHz closed-loop, and the robot’s states were sampled at the same frequency. These sampled states were saved in the states buffer; the time-series states were fed into the trained network as input, and the output of the network is estimated GRF. The results of the naive estimation method are shown in Fig. 7B.

In the naive Sim2Real approach, the RMSE was 8.2118 N on the given robot motion. This performance was attributed to the domain gap, which included mechanical features such as gear backlash, static friction, torsional stress, and unmodeled wires for communication and power distribution. Despite the GRF estimator’s unsatisfactory performance, the estimated GRF still followed the overall profile of the measured GRF, which indicated that the peak values occur simultaneously. In other words, the trained network learned the relationships between the time-series robot states and GRF but had nonlinear offsets.

We propose using an additional neural network directly after the GRF estimation network to bridge the domain gap in order to overcome the domain gap between the simulation and the physical world.

Sim2Real bridge network

To train an additional network to bridge the estimated GRF to the physical world, we first measured the GRF from the physical world using a load cell.

Figure 8 shows the overall framework from the data collection process to the training process of the Sim2Real bridge network. Estimated GRFs were collected as the inputs of the Sim2Real bridge network from the GRF estimation network. The measured forces were collected to use as ground-truth for the output of the Sim2Real bridge network. A total 131,968 datasets were collected in nine different sinusoidal motions. The entire process of collecting real-world datasets was completed in three minutes.

The physical robot’s state contained noise components caused by sensors such as encoders, which were for measuring joint positions and velocities. In addition, delays exist in the physical world, which are caused by communication between PC and motor drivers, while the simulation has no delay induced by communication. Because of the noise and delay in the inputs of the GRF estimation network, the estimated GRF also contains noise components.

To address these hindrances in the training process of the bridge network, we decided to use time-series estimated GRF as input to teach system characteristics to the bridge model. In addition, we determined the amount of time-series data that was needed to teach the model the system characteristics. For training the Sim2Real bridge network, the hierarchy of the MLP model was fixed with six hidden layers and eight nodes, utilizing the swish activation function. The Adam optimizer was employed to update the weights of the MLP model. To determine the appropriate time-series input, we conducted experiments with the different combinations of interval time and the number of time-series.

A total of 20 combinations of time-series inputs were tested, and the training results are shown in Table 5. In the models that used five time-series inputs with different interval times, MSE decreased when the interval time increased. However, in the models that used 20 time-series inputs, the best case was when the interval time was 20 msec. Likewise, in the cases with the same interval time but different numbers of time-series inputs, the tendency of MSE as similar to the previous case. The best-case occurred when the number of time-series inputs was 50 and the time interval was 10 msec. In order to lower the number of the inputs for the fast estimation of the model, a model that uses 20 number of time-series and 20 msec time interval was selected for the Sim2Real bridge network. (The performance of the selected model is shown in bold in Table 5).

Process for GRF estimation

This section presents the process for estimating GRF in a point-foot robot in the physical world. The overall block diagram for GRF estimation is shown in Fig. 9A. The motion controller generates control inputs in a 1 kHz feedback loop for the given motion, and the robot’s states were saved to the states buffer for generating time-series states for the GRF estimation network in the two-staged GRF estimator. The output of the GRF estimation network was also saved to the buffer and used as the time-series inputs of the Sim2Real bridge network. Finally, the Sim2Real bridge network estimated the physical world’s GRF of the robot. The proposed two-stage neural network’s GRF estimation performance in the physical world is shown in Fig. 9B and was compared with the observer-based method. Table 6 provides a comparison and evaluation of the GRF estimation performance using the proposed method with the observer-based method. The evaluation includes two metrics, RMSE and R-squared. RMSE resulted in a lower value at 1.2094, and R-squared had a value closer to 1 at 0.8791, indicating that the proposed method had a better estimation performance than the observer-based method.