Pilot study of the relation between various dynamics of avatar experience and perceptual characteristics

System

The avatar system developed by this study consists of a human elbow joint motion control dynamics model (Neuromusclularskeletal model: nMSS model) based on the equilibrium point hypothesis, which claims that the adjustment of two parameters, namely, the equilibrium point of the body and the joint stiffness, is important in human motion control (Matsui et al., 2014; Matsui et al., 2015; Nagai et al., 2019; Matsui et al., 2022). In this study, the equilibrium point movement is one-degree-of-freedom motion of the human elbow joint in the horizontal plane, which corresponds to the joint angle. AA ratio r and agonist–antagonist muscles sum (AA sum) s, which are defined as the contributions to the joint stiffness, are expressed by Eqs. (1) and (2), where m_e is the percent maximum voluntary contraction(%MVC)(Vera-Garcia, Moreside & McGill, 2010), which is normalized EMG obtained from EMG during maximal voluntary contraction in muscle alone, for the extensor side and m_f is %MVC for the flexor side. (1)${\displaystyle r=\frac{m_{\mathrm{e}}}{m_{\mathrm{f}}+m_{\mathrm{e}}}}$ (2)${\displaystyle s=m_{\mathrm{f}}+m_{\mathrm{e}}}$

In this study, the NMSS model is applied to the human elbow joint control by EMG and is represented as a transfer function cascade coupling of the “neuromuscular system (second-order delay + dead time) + musculoskeletal system (second-order delay)” (Fig. 1), wherein each transfer function is expressed by Eqs. (3) and (4), where G_NM(s), K_NM, ω_nNM, ζ_NM, and L_NM are the transfer function, gain, natural angular frequency, damping coefficient, and dead time of the neuromuscular system, respectively, and G_MS(s), K_MS, ω_nMS, ζ_MS, and L_MS are the transfer function, gain, natural angular frequency, damping coefficient, and dead time of the musculoskeletal system respectively. (3)${\displaystyle G_{\mathrm{NM}}\left(s\right)=K_{\mathrm{NM}}\cdot \frac{{\omega }_{\mathrm{nNM}}^2}{s^2+2{\zeta }_{\mathrm{NM}}{\omega }_{\mathrm{nNM}}s+{\omega }_{\mathrm{nNM}}^2}\cdot e^{-L_{\mathrm{NM}}s}}$ (4)${\displaystyle G_{\mathrm{MS}}\left(s\right)=K_{\mathrm{MS}}\cdot \frac{{\omega }_{\mathrm{nMS}}^2}{s^2+2{\zeta }_{\mathrm{MS}}{\omega }_{\mathrm{nMS}}s+{\omega }_{\mathrm{nMS}}^2}}$

This avatar system can control virtual limbs “continuously and in real time” by utilizing NMSS model-based rather than EMG triggers (Kaneko, 2016). When using EMG as a trigger, a considerable time delay from the actual “intention” can occur based on the threshold setting. Furthermore, it only enables the realization of a constant avatar motion in accordance with the threshold value, lacking the capability to “intentionally stop joint motion at the intended position” like the NMSS model. Therefore, the NMSS model was adopted in this study. It has been shown that the human NMSS model parameters change according to the AA sum (Iimura et al., 2011; Gong et al., 2020; Matsui et al., 2022). Therefore, it is unlikely that humans sustain the AA sum s constant during exercise, and when EMG is input to the NMSS model, it is desirable to sequentially change the model parameters according to the AA sum s obtained from the EMG. However, for the sake of simplicity, at this stage, the NMSS model is regarded as a time-invariant model for the avatar system construction. In the experiment, EMGs (m_e, m_f) of muscles having opposing actions on the participant’s right elbow joint are obtained, and the AA ratio r expressed in Eq. (1) is input to the NMSS model in a PC to calculate the joint angle, which is reflected in an upper limb avatar placed on a VR with a head-mounted display (HMD, VIVE Pro, HTC Corporation). The avatar shoulder placement matches that of a real body captured by 3D motion capture Azure Kinect DK (Microsoft Corporation, Redmond, WA, USA).

The system configuration is shown in Fig. 2. In the experiment, the EMG of the extensor and flexor muscles, which are triceps brachii and biceps brachii, respectively, were obtained from electrode pads attached to the participant’s upper arms, and after analog digital (AD) conversion, the data were processed in the PC. For the EMG acquisition, WEB-5000 (NIHON KOHDEN CORPORATION) was used in the preliminary experiments and EBM-102 (Unique Medical Co., Ltd., Tokyo, Japan) was used in the main experiments. Changes in equipment during the course of the study were due to changes in laws and regulations in Japan (Telecommunications Bureau of the Ministry of Internal Affairs and Communications of Japan, 2005). AD conversion and data acquisition were performed with Labview2016 (NI) on a PC, combined with a data acquisition (DAQ) board. When using WEB-5000, EMG were acquired using a 10 Hz high-pass filter and a 100 Hz low-pass filter. Meanwhile, When using EBM-102, a 10 Hz high-pass filter and a 120 Hz low-pass filter ware used. Then, these filters were applied to the EMG acquired at a sampling frequency of 1000 Hz, and after rectification, a 22 Hz low-pass filter was applied and converted to %MVC (m_e, m_f).

The elbow joint angle calculated angular displacement Δθ(t) from the standard position is continuously reflected by 30 FPS on the upper limb elbow joint avatar in VR space in realtime. The standard position was set at 90°. Therefore, all trial experiments start from the real elbow joint at about 90°. With this system, the participant manipulates the virtual upper limb elbow joint avatar (Fig. 3) from the standard position using the EMG of the extensor and flexor muscles. The results are then visually displayed through VR. This means that the participant flexes and extends the virtual upper limb elbow joint avatar in VR space, which participants perceive their own arm with dynamics different from their respective physical dynamics (NMSS).

Experiment

In this study, the experiment was conducted twice: preliminary experiment and main experiment.

The experiments involving participants in this study were conducted with the approval of the Ethics Committee on Research Involving Human Participants (R3-3-1) of the Graduate School of Engineering Science, Osaka University. Written informed consent was obtained from the participants.

Preliminary experiment

In a preliminary experiment, this study investigated how a change in SoO, SoA, and CL by manipulating avatar settings, specifically the natural angular frequency and dead time. In addition, discrimination threshold investigation was conducted, determining whether the avatar was perceived as displaying “lack of movement” or “hyper responsive movement” when settings were altered, details are provided in the Appendix. To validate the fundamental functions of the avatar, parameters unrelated to delay, such as gain and damping coefficient, were also examined and detailed in Appendix. For the research participants, six healthy participants (23.2 ± 0.7 years) wore HMD and experienced an upper limb avatar by performing the following TASK.

TASK: In VR space, the user rapidly repeats flexion and extension movements of the virtual upper limb elbow joint avatar five times in arbitrary timing at 90°and 150°which are remembered in VR space before TASK. If the user cannot flex the elbow at 90°or 150°, the user flexes it at the maximum or minimum angle that can be moved at that time.

Participants ware seated in a chair with their upper arm constrained to the platform and their forearm fixed to another platform which is movable in horizontal plane. Therefore, respective participants could move their elbow joint without any shoulder movement and gravitational pull. There were “comparison settings”: natural angular frequency [rad/s] (ω_npi = ω_nNM = ω_nMS) is { ω_np1, ω_np2, ω_np3, ω_np4}= {8, 6, 4, 2}, dead time [ms] (L_i) is { L₁, L₂, L₃, L₄}= {0, 50, 150, 300}, and “standard setting”: gain [ − ] (K_st = K_NM⋅K_MS) is 300, natural angular frequency [rad/s] (ω_npst = ω_nNM = ω_nMS) is 6, damping coefficient [ − ] (ζ_st = ζ_MN = ζ_MS) is 0.7, dead time [ms] (L_st) is 0. Immediately after the standard setting a set of three comparison settings were experienced in sequence (in only final phase, two comparison settings were experienced). The comparison settings order was randomized so that participants experienced comparison settings twice. Comparison settings are combinations of gain, natural angular frequency, damping coefficient, and dead time, each of which varies from the standard setting while the other parameters are constant. From this protocol, the dummy comparison settings is the same as the standard setting which occurs randomly.

After experiencing each of the comparison settings, participants answered each evaluation questionnaire orally while wearing the HMD. The following four questionnaires were administered. Three types of questionnaires were used to rate the “SoO (how much you felt the avatar is your own body)”, the “SoA (how much you felt you were controlling the avatar yourself)”, and the “CL (how much comfort you have when controlling the avatar)” on a worst to best scale of “ −3 to +3” compared to the standard setting (0) in “0.25” increments. These experimental protocols is shown in Fig. 4.

Examples of system response

Examples of the avatar system’s responses to the NMSS model inputs are shown in Fig. 5. It also plots the output of the following sigmoidal function input for each of the comparison settings in the preliminary experiment and main experiment. (5)${\displaystyle \Delta r=\frac{1}{1+exp\left(\frac{20}{3}\cdot \left(t-0.5\right)\right)}}$ where Δr is an input that simulates the AA ratio with an initial value of 0 and t denotes time. As indicated in the figure, natural angular frequency contributes to phase delay, and dead time contributes to time lag, respectively. Figure 5 plots the output of the actual AA ratio input versus the real elbow joint angle at that time in some main experiment comparison settings; as indicated in the figure, lower natural angular frequency contributes to negative delay and makes some small noise because AA ratio is directly reflected to the avatar joint angle. Settings other than the comparison settings shown in those legends follow the standard setting.

Preliminary experiment

Natural angular frequency

The evaluated values and effect sizes at natural angular frequency are shown in Fig. 6 and Table 1.

Table 1:

Natural angular frequency in the preliminary experiment: effect sizes (upper: SoO, middle: SoA, and lower: CL).

	ωnp2	ωnp3	ωnp4
ωnp1	.38*	.70**	.41*
ωnp2		.64**	.51**
ωnp3			.73**

	ωnp2	ωnp3	ωnp4
ωnp1	.07	.15	.62**
ωnp2		.27	.68**
ωnp3			.72**

	ωnp2	ωnp3	ωnp4
ωnp1	.00	.40*	.53**
ωnp2		.61**	.70**
ωnp3			.86**

DOI: 10.7717/peerjcs.2042/table-1

• SoO values peaked at ω_np3 and fell at ω_np4. The effect sizes of ω_np1 − ω_np2 and ω_np1 − ω_np4 were “medium”, and all other effect sizes were “large”.

• SoA values went down to ω_np4. The effect sizes of ω_np1 − ω_np4, ω_np2 − ω_np4, and ω_np3 − ω_np4 were “large”.

• CL values showed a peak at ω_np3 and a drop at ω_np4. The effect size of ω_np1 − ω_np3 was “medium”, while that of ω_np1 − ω_np4, ω_np2 − ω_np3, ω_np2 − ω_np4, and ω_np3 − ω_np4 were “large”.

Dead time

The evaluated value and effect size at dead time are shown in Fig. 7 and Table 2.

Table 2:

Dead time in the preliminary experiment: effect sizes (upper: SoO, middle: SoA, and lower: CL).

	L2	L3	L4
L1	.13	.38*	.52**
L2		.33*	.52**
L3			.58**

	L2	L3	L4
L1	.13	.15	.69**
L2		.02	.67**
L3			.67**

	L2	L3	L4
L1	.79**	.12	.82**
L2		.58**	.96**
L3			.43*

DOI: 10.7717/peerjcs.2042/table-2

• SoO values showed a small peak at L₃ and a drop at L₄. The effect sizes of L₁ − L₃ and L₂ − L₃ were “medium”, while those of L₁ − L₄, L₂ − L₄, and L₃ − L₄ were “large”.

• SoA values were lower at L₄. The effect sizes of L₁ − L₄, L₂ − L₄, and L₃ − L₄ were “large”.

• CL values peaked at L₂ and decreased as the comparison setting value increased to L₃ and L₄. The effect size of L₃ − L₄ were “medium”, while that of L₁ − L₂, L₁ − L₄, L₂ − L₃, and L₂ − L₄ were “large”.

Main experiment

The evaluated values and effect sizes at natural angular frequency are shown in Fig. 8 and Table 3.

• SoO values increased with decreasing values in the range ω_nm1 ≥ ω_nmi ≥ ω_nm4. Meanwhile, ω_nm4 ≥ ω_nmi ≥ ω_nm5 did not change much. ω_nm5 ≥ ω_nmi ≥ ω_nm6 increased with decreasing values. The value increased as the value decreased. The effect size of ω_nm4 − ω_nm6 and ω_nm5 − ω_nm6 showed “medium”, all others showed “large”.

• SoA values increased with decreasing values in the range ω_nm1 ≥ ω_nmi ≥ ω_nm4. Meanwhile, ω_nm4 ≥ ω_nmi ≥ ω_nm5, did not change much. ω_nm5 ≥ ω_nmi ≥ ω_nm6 decreased with decreasing values. The values decreased with decreasing values. All effect sizes except ω_nm4 − ω_nm5, ω_nm4 − ω_nm6, and ω_nm5 − ω_nm6 showed “large”.

• CL values increased with decreasing value in the range ω_nm1 ≥ ω_nmi ≥ ω_nm4 and decreased with decreasing value in the range ω_nm4 ≥ ω_nmi ≥ ω_nm6. In other words, it peaked at ω_nm4. All effect sizes except ω_nm4 − ω_nm5, ω_nm4 − ω_nm6, and ω_nm5 − ω_nm6 showed “large”.

Distribution in the low natural angular frequency domain

Figure 9 shows the distribution of peak evaluations of natural angular frequency in the low range for each research participant. It can be seen that the preliminary experiment peak is closer to the high frequency region compared to the main experiment. In other words, in this experiment, the low frequency region distribution of the natural angular frequency changed.

Preliminary experiment

SoA at dummy setting

In natural angular frequency and dead time, SoA was higher than 0 in the dummy setting, which is the same case as in the standard setting. This suggests that SoA may be improved by repetition compared to the other perceptual characteristics, since the participants always experience the avatar in each setting after experiencing the standard setting due to the experimental protocol. In contrast, CL is close to 0 in the dummy setting. CL is introduced as an alternative parameter to limb heaviness to encompass inquiries not only about the sensation of heaviness but also about the sensation of lightness. This additional information by reports suggest a relation between comfort and SoA (Mishima et al., 2023). SoA is changed from this result and CL may be a more acute questionnaire method in some cases.

Summary of preliminary experiment

In the preliminary experiment, we verified that similar results to those observed in the previous study could be replicated for positive delays using the settings of natural angular frequency and dead time. Simultaneously, we gained new insights into relationship between each parameter and perception through this avatar experience system. Moreover, we affirmed the validity of main experiment, as the setting range explored in the preliminary experiment was deemed insufficient for confirming the perceptual characteristics in negative delay, which is the purpose of this study.

Main experiment

SoO increased with decreasing natural angular frequency values in the range of ω_nm1 ≥ ω_nmi ≥ ω_nm4. SoO did not change much in the range of ω_nm4 ≥ ω_nmi ≥ ω_nm5. In the range of ω_nm5 ≥ ω_nmi ≥ ω_nm6, SoO increased with a decreasing natural angular frequency value. In addition, looking at the effect sizes for ω_nm1 − ω_nm2, ω_nm2 − ω_nm3, ω_nm3 − ω_nm4, ω_nm4 − ω_nm5, and ω_nm5 − ω_nm6, which compare adjacent comparison settings, respectively show “large”, “large”, “large”, “slight”, and “medium”. Meanwhile, ω_nm4 has “medium” effect to ω_nm6.

SoA increased with a decreasing natural angular frequency value in the range of ω_nm1 ≥ ω_nmi ≥ ω_nm4. SoA did not change much in the range of ω_nm4 ≥ ω_nmi ≥ ω_nm5. In the range of ω_nm5 ≥ ω_nmi ≥ ω_nm6, SoA decreased as the natural angular frequency value decreased. Looking at the effect size of ω_nm1 − ω_nm2, ω_nm2 − ω_n3, ω_nm3 − ω_nm4, ω_nm4 − ω_nm5, and ω_nm5 − ω_nm6, which compare adjacent comparison settings, respectively show “large”, “large”, “large”, “slight”, and “slight”. Meanwhile, ω_n4 has “medium” effect to ω_nm6. Thus, the downward trend of SoA in the range of ω_nm4 ≥ ω_nmi ≥ ω_nm6 is hardly effective. However, in the range of ω_nm1 ≥ ω_nmi ≥ ω_nm4, the tendency of SoA to increase is effective.

CL increased with a decreasing natural angular frequency value in the range of ω_nm1 ≥ ω_nmi ≥ ω_nm4 and decreased with a decreasing value in the range of ω_nm4 ≥ ω_nmi ≥ ω_nm6. Looking at the effect sizes for ω_nm1 − ω_nm2, ω_nm2 − ω_nm3, ω_nm3 − ω_nm4, ω_n4 − ω_nm5, and ω_nm5 − ω_nm6, which compare adjacent comparison settings, respectively show “large”, “large”, “large”, “small” and “slight”. Thus, in the range of ω_nm4 ≥ ω_nmi ≥ ω_n6, the downward trend of CL is hardly effective. However, in the range ω_nm1 ≥ ω_nmi ≥ ω_nm4, the upward trend of CL is effective.

This experiment is an additional experiment based on the preliminary experiment results. As noted earlier, natural angular frequency is a parameter of delay. These results indicate that a high natural angular frequency, which causes a smaller delay or negative delay, has the possibility to makes SoO, SoA, and CL decrease similar to a large delay. In a previous study, EPA trial without “intention” was only discussed in that it makes no SoA. In this study, it is indicated that SoA decreases gradually by decreasing delay. This delay has the possibility to be negative and is based on voluntary signal. Therefore, this situation can be called EPA trial with “intention”. Meanwhile, CL and SoO have a similar trend with SoA. Since increasing natural angular frequency makes some small noise, this point has to be investigated by using avatar with intentional noise and not rely on natural angular frequency in future studies. In contrast, a low natural angular frequency area similar to that in the preliminary experiment has a different trend than that in the preliminary experiment, i.e., there is no decreasing trend at L₄. It is considered that it is caused by distribution of perceptual characteristic distribution for each participant.

Distribution in the low natural angular frequency domain

The histograms of Fig. 9 show that the distribution of the peaks in the evaluation at natural angular frequencies is different for each individual. This suggests that peaks in the low frequency range are different for each individual. These results might mean that the natural angular frequency value appropriate for each individual is different, and those dynamics around these peaks are close to those corresponding to the internal model of each individual participant. Hence, the histograms of the peaks in the preliminary experiment and main experiment suggest the potential for personalized peaks unique to each individual. This observation opens the possibility for evaluating an internal model that has not been realized thus far.