AI-driven balance evaluation: a comparative study between blind and non-blind individuals using the mini-BESTest

DTW and KNN classification

As previously stated, the first classification task was based on the similarity of the signals using DTW and the classification with KNN. The DTW algorithm was computed by comparing each axis (X, Y, Z) of each of the 12 sensors between all the participants. Thus, the matrix of distances evaluated had $3\ast 12\ast \left(\begin{array}{c}29\\ 2\end{array}\right)=14\text{,}616$ comparisons on each leg. It was used the fastdtw Python library to compute the DTW distance. The KNN algorithm was used to classify the participants according to the DTW distance.

Following an exploratory analysis, the sensors that were most relevant for the classification were selected. The sensors selected for this purpose included the pelvis, right foot, left foot, right hand, left hand, and head. The duration of the recording was also included as a feature. No other sociodemographic information was included. A sequential forward feature selection with a KNN estimator ( $k=3$) and the 3-CV method was used to select the most relevant features. The value $k=3$ was chosen because it is the default parameter in the sklearn Python library and is appropriate given the limited number of participants, as larger values would dilute the classification performance. Feature subsets of 20% and 50% were tested to balance the reduction of dimensionality with the retention of information, with no significant differences in the results, demonstrating the robustness of the model. After feature selection, the DTW data were standardized. A KNN classification was performed with $k=[1:28]$ neighbors, covering the feasible range based on the number of participants. The classification was performed using the Euclidean distance.

Time features and random forest classification

The second classification task was based on four time features extracted from the accelerometer signals and classification with Random Forest (refer to Fig. 2). We extracted 144 features including: mean, standard deviation, energy, and amplitude for each axis of the 12 sensors. The duration of the recording was also added as a feature. No other sociodemographic information was used. The features were computed using an overlap sliding window. Several combinations of window sizes $ws=[0.5,1,1.5]\mathrm{s}$ and overlap times $ot=[0,0.5]\mathrm{s}$ were studied. Window sizes smaller than 0.5 s were avoided since the movements of interest are not rapid enough to justify shorter windows, while window sizes larger than 1.5 s were excluded because some activities are brief, and larger windows could reduce classification performance by averaging out relevant information. The overlap times were selected to align with the range of window sizes tested, ensuring sufficient data coverage while avoiding redundancy.

A sequential forward feature selection with the same parameters as in the DTW experiment was used to select the most relevant features. After that, the Random Forest algorithm was used to classify participants according to the selected time features. The Random Forest algorithm was implemented using the sklearn Python library. A grid search was performed to find the best hyperparameters for the Random Forest algorithm. The hyperparameters tuned were: number of estimators $ns=[25,50,500,1\text{,}000]$, maximum depth $md=[10,20,30]$, minimum sample split $mss=[2,5,10]$, and minimum sample leaf $msl=[2,5,10]$. The range of values for the number of estimators $ns$ was chosen based on standard practices for datasets of this size, as these values balance computational efficiency and model performance.

The windowing led to a data set in which different participants had different numbers of samples. For training, each window was assigned the same label as the participant. In the testing phase, only one label per participant was used. To achieve this, the final prediction was selected as the most prevalent label for each participant.

Deep learning classification

The third classification task was based on raw signals and classification with a DL model (Fig. 2). The DL models were fed alternatively with two kinds of raw signals: (i) the raw acceleration signals from the IMUs, and (ii) the quaternions obtained from the IMUs. A quaternion is a mathematical representation of a rotation in a three-dimensional space. The quaternions were obtained from the acceleration and gyroscope data using the Mahony filter to estimate the orientation of the sensor. The Attitude and Heading Reference Systems (AHRS) Python library was used to compute the quaternions.

Both raw signals were windowed and standardized using the Standard Scaler function from the sklearn Python library. The signals were segmented using a sliding window technique (Reyes Leiva, Gato & Olmedo, 2023). The window sizes used were $ws=[0.5,1]\mathrm{s}$ with fixed overlap time $ot=20\mathrm{m}\mathrm{s}$ to do the data augmentation necessary for these kinds of models.

Four DL architectures were tested: CNN, LSTM, GRU, and two architecture variations of a CNN-LSTM hybrid model. Table 2 shows the configuration of the models’ architectures. These network architectures were chosen at the discretion of the experimenter after a number of iterations of exploratory analysis. All models were trained with: batch size $=[32,64]$, epochs $=[25,50,100]$, learning rate $=0.001$, optimizer $=Adam$, $ReLU$ as activation function, and loss= categorical crossentropy. The models were implemented using the keras Python library.

It is worth mentioning that in the exploratory analysis time-series data augmentation was employed to correct the imbalance of the dataset. In order to achieve this objective, the raw signals were augmented with different techniques, such as time change, time warping, and amplitude modulation, using the tsaug Python library. However, the results were not improved and the data augmentation was discarded. A total of 20% of the participants were used for validation and 20% for testing. Due to the reduction in the dataset resulting from the validation split, in this experiment a double nested 5-CV strategy was used. The models were trained using the keras Python library.

Clustering based on sociodemographic information and mini-BESTest scores

The first cluster analysis was based on sociodemographic information and the mini-BESTest scores. A total of 23 features were used as input data: (i) the 14 individual punctuations and four aggregate punctuations of the mini-BESTest, and (ii) sociodemographic information on sex, age, height, weight, and sport practice. Before the cluster analysis, a feature selection was conducted by eliminating first high correlated features —those with absolute correlation higher than 0.9—, and then an RFE with linear regression approach was used. The features were standardized before the cluster analysis.

Clustering based on IMU data

The second cluster analysis was based on time features. On the 144 features extracted from the IMU data, a feature selection was carried out, as in the previous cluster analysis, by eliminating first the high correlated features and then using RFE. The features were also standardized.

Expert evaluation of Mini-BESTest

In this section, it was analyzed whether mini-BESTest is capable of identifying differences in balance between blind and sighted individuals. Figures 3A and 3B depicts the punctuation histograms for each of the fourteen activities of the mini-BESTest for blind and sighted individuals.

As can be seen in Fig. 3, the punctuation of some activities was unable to identify differences between blind and sighted individuals, but some of them did. To better understand the differences in balance between blind and sighted individuals, a statistical analysis was performed.

The results shown in Table 3 suggest that there were no significant differences between blind and sighted individuals in the reactive postural control and sensorial orientation categories. However, there were significant differences in the categories of anticipatory postural control and dynamic gait, which is in line with the results of Wiszomirska et al. (2013) for visually impaired women. The overall mBESTest score was also significantly different between blind and sighted individuals, but mainly due to differences in the categories of anticipatory postural control and dynamic gait.

The average mini-BESTest score for all participants was $25.9\pm 2.3$, leaving $86.7\mathrm{\%}$ of sighted individuals and $28.6\mathrm{\%}$ of blind individuals above the average. These results match the assumptions made in the sample size calculation. With these results, we can also say that the average balance of young sighted adults is $27.3\pm 1.2$, while for blind individuals it is $24.7\pm 2.1$. These insights can be used to compare the expected results with other cohorts of participants such as those described in Almeida, Marques & Santos (2017).

To discern which of the activities in the anticipatory postural control and dynamic gait categories are responsible for the differences between blind and sighted individuals, a post hoc analysis was performed. The values are shown in Tables 4 and 5.

The evaluation of anticipatory postural controls involves the performance of tasks that necessitate deliberate and volitional movements that challenge the subject’s ability to maintain equilibrium. Include Tests 1, 2, and 3, which correspond to the following tasks: sit to stand, rise to the toes, and stand on one leg. The mini-BESTest employs these tasks to assess an individual’s capacity to anticipate and prepare for postural alterations.

Assessment of dynamic gait entails the performance of tasks that challenge the individual’s capacity to maintain balance while walking under varying conditions. This assessment involves activities such as those tested in Tests 10, 11, 12, 13, and 14, including: changing speed, walking with head turns, walking with pivot turns, stepping over obstacles, and timed up and go with dual task. The mini-BESTest assesses the individual’s capacity to manage these dynamic challenges, which are essential for safe and effective ambulation in daily life.

The results of the post hoc analysis in Tables 4 and 5 showed that the differences between blind and sighted individuals in the anticipatory postural control category were mainly due to Test 3. The differences in the dynamic gait category were mainly due to Test 11 and Test 14. The results suggest that the anticipatory postural control and dynamic gait categories were the most relevant to discern between blind and sighted individuals.

The results suggest that blind individuals have more difficulty maintaining balance while standing on one leg compared to sighted individuals. The total number of sighted individuals was evaluated with the highest value in Test 3, indicating a good balance. In contrast, blind individuals exhibited a spectrum of balance, including poor, mild and optimal balance: 28.6% of blind individuals had poor balance, 42.9% had mild balance, and 21.4% had good balance.

These results satisfied the objective (i) of this study. The mini-BESTest identified differences in balance between blind and sighted individuals. The differences were mainly due to the anticipatory postural control and dynamic gait categories, which were the most relevant to discerning between blind and sighted individuals.

Henceforth, the focus will be directed towards the Test 3 activity, as it is the most relevant to evaluate anticipatory capacity, where health professionals can concentrate their attention.

DTW and KNN classification analysis in Test 3

The analysis of DTW distances between accelerometer signals of the different subjects aims to show if there is a significant difference between the signals of the subjects in general and according to the score in Test 3. In the statistical analysis of the DTW distance matrix, some interesting results stood out.

To perform a comparison of the distances between the signals of the subjects, the mean distance between the signals of the subjects was calculated. It is defined as the arithmetic mean of the distances of each acceleration axis of a subject to the rest of the subjects. The comparison was made by punctuation. The results are shown in Table 10.

The difference in means between the distances of the signals of subjects with a score of 1 and 2 was 331.73 $\pm$ 95.96, while the difference in means between the distances of the signals of subjects with a score of 2 was 367.87 $\pm$ 82.17. The comparison of these two distributions showed a significant difference with $p<0.00001$. This means that there were differences between the signals of the subjects with a score of 1 and 2. In contrast, the fact that the remaining comparisons were not significant indicates that there were no significant differences between the signals of subjects with a score of 0 and 1 and between subjects with a score of 1 and 2, which may be symptomatic of the signals being more similar to each other and the classification being more complicated.

Going deeper into this comparison, and looking at Table 10, it can be asserted that it was in blind subjects where the distance between signals with a score of 1 and 2 (288.45 $\pm$ 85.05) was lower than in sighted subjects (427.28 $\pm$ 101.16). This again reveals that, in general, the signals of blind subjects with a score of 1 and 2 were more similar to each other than those of sighted subjects, which may also be a factor that affects the classification.

Other more detailed analysis suggested that the signals of only five subjects with vision ( $33.3\mathrm{\%}$) had significant differences from other subjects and that seven blind subjects, who constituted 50% of the population, did not. This will also be a handicap for the classification.

As expected from the aforementioned complications, the classification of the signals of the subjects according to their score in Test 3 was tough. A KNN classification analysis was conducted and the best results are shown in Table 11. A weighted F1-score of 73.9 $\pm$ 9.3 was achieved for binary classification grouping bad and mild evaluations. This model exceeded the ground truth—which is 66.7—, however, it was not significantly different from the next best model for $k=4$ and F1-score 68.9 $\pm$ 15.9 ( $p=0.4082$). The best result was obtained for a selection of 12 sensors: head, neck, torso, pelvis, left and right forearm, hands, shanks, and feet.

Time features and random forest classification analysis in Test 3

Table 11 shows the best results of the classification analysis using time features and Random Forest. A weighted F1-score of 70.0 $\pm$ 13.6 was achieved for binary classification grouping bad and mild evaluations, which slightly exceeded the ground truth, which is 66.7. However, the difference was not significant ( $p=0.3069$) from the next best model with an F1-score of 62.0 $\pm$ 10.3. The best result was obtained for a selection of 24 features belonging to four sensors: the left and right shanks. It is also noteworthy that none of the selected features includes the energy of the signal.

As expected, the imbalance and errors in the evaluations lead to a decrease in the classification performance and lead to the application of a binary classification approach to improve the results. The result was lower than the DTW + KNN model, despite the fact that the difference is not significant ( $p=0.5535$).

Deep learning classification analysis in Test 3

In this section, the classification analysis of Test 3 with Deep Learning and raw acceleration signal is shown. As mentioned in the methodology, different architectures, input data such as quaternions, and temporal windows was tested. The results of the best combination of architecture and input data are shown in Table 11.

The best result was obtained with a weighted F1-score of 85.6 $\pm$ 6.7 for binary classification grouping mild and good evaluations. This model exceeded the ground truth, which is 66.7, and the difference is significant ( $p=0.0272$) from the next-best model, which is DTW + KNN. However, the best result was obtained for the entire set of sensors. As a result, for the current results it was not possible to determine which sensors were most relevant for the classification of Test 3 and remove any of them.

Because the best model combines mild and good evaluations, in contrast to other models, it was evident that mild evaluations were the most difficult to classify. Nine out of thirteen incorrectly evaluated by duration were mild evaluations that should have been good or good evaluations that should have been mild.

The input data was the acceleration, as the quaternions did not improve the results of acceleration in any of the cases. The results of options 1 and 2 were similar despite the variation in the temporal window. The F1-score obtained was 85.6 $\pm$ 6.7, and exceeded the ground truth, which was 66.7.

The difference in F1-score with the DTW + KNN model was 11.7, which was statistically significant ( $p=0.0272$). Although the distance with the RF model was 15.3, the large standard deviation of this model made the difference significant by a very small margin ( $p=0.0540$). Based on these results, we can confirm that the Deep Learning model was the best for classifying Test 3.

It was demonstrated by a range of training exercises that employ different sensor configurations that the most favorable outcome was achieved with a six-sensor configuration when executing Test 3. The positioning of these sensors should be as follows: pelvis, left and right hands, left and right feet, and head.

Cluster analysis based on sociodemographic information and mini-BESTest scores

First, Fig. 5 shows the correlation matrix of the features. As can be seen, all activity tests of the ‘Sensorial Orientation’ category, ‘Test 1’, and ‘Test 10’ did not have variation, thus providing no information to the clustering algorithm, and the six features were discarded.

Furthermore, by removing features with absolute correlation higher than 0.9, five more features were removed: ‘Anticipatory’, ‘Test 4’, ‘Test 5’, ‘Test 6’ and ‘Reactive Postural Control’. Twelve features remained. The RFE selected the most relevant features for the cluster analysis. Despite the fact that ‘Test 3’ was the most relevant feature and a good clustering metric was achieved with only this feature, the best clustering metrics were achieved with three features: ‘Test 3’, ‘Age’ and ‘Sex’.

Figure 6 shows the results of the cluster analysis for the best feature and the three best features after recursive feature elimination, respectively. The results showed that the optimal number of clusters was 2 for all metrics and set of features. It is worth mentioning that ‘Test 3’ is the most relevant feature and good clustering metrics are achieved only with this feature. However, the best clustering metrics are achieved with three features: ‘Test 3’, ‘Age’, and ‘Sex’.

The clustering metrics were not limited and there was no level to compare with and, despite the fact that the clustering metrics were not high, Fig. 7 highlights that the participants were clearly separated into two groups, which corresponded to the original blind/sighted groups with a 90% F1-score (only one participant was misclassified).

Thus, the cluster analysis based on sociodemographic information and mini-BESTest scores suggested that there were two groups of participants: blind and sighted. An interesting aside result of this post hoc analysis was that a person can be classified as blind or sighted based only on the result of the ‘Test 3’, their ‘Age’, and ‘Sex’.

Cluster analysis based on IMU data

A cluster analysis based on IMU data was performed to determine whether the evaluations of the activities can be grouped into more than two or three groups (scores). The results of the cluster analysis are shown in Fig. 8. The best cluster analysis results were achieved with two features: ‘torso_ay_mean’ and ‘torso_ay_energy’. The results showed that the optimal number of clusters was 2 for all the WCSS, Shilhouette, and Davies-Bouldin metrics.

These results indicated that time features based on acceleration signals were unable to group the evaluations of the activities into more than two scores, which was consistent with the results of the mini-BESTest evaluation.

It is worth mentioning that the ‘duration’ feature was then the next feature that appeared in the next best clustering, which was indicative that the duration of the activities was an important factor in the evaluation of the activities as expected. What is more interesting is that the energy of the torso acceleration signal was one of the most relevant features, indicating that the energy of the torso acceleration signal was an important factor in the evaluation of the activity ‘Test 3’. However, this sensor was only selected in the DTW + KNN model and not in the RF model. This could be due to the fact that the torso was important, but its low variability was good for clustering, but not for classification. In fact, the subsequent features that appear are related to the foot and shank, which are the most relevant sensors for the classification of activity ‘Test 3’ in the RF model.

Statistical analysis of mini-BESTest

After the evaluation of all participants, a post hoc analysis was performed to find differences between sighed and blind people. As a result, the highest difference was found in Test 3, which evaluates the ability to maintain balance while standing on one leg. This result was also pointed out by Rutkowska et al. (2015) for visually impaired children, and also stated by Blomqvist & Rehn (2007). In contrast, as indicated by Kümmel et al. (2016), this generic one-leg stance test is specific enough to identify differences between blind and sighted individuals. However, as the authors suggest, each study needs its own specific test to evaluate balance, and this might be the case for the present study.

These findings emphasize that standing on one leg is a critical exercise for differentiating between blind and sighted individuals, as it directly measures static balance capacity. This insight supports the idea that focusing on this specific test could streamline the assessment process by reducing the need to perform the entire mini-BESTest. This approach aligns with the findings of Pennell et al. (2024), who demonstrated that a reduced version of the mini-BESTest that focusses on key elements, including static balance tasks, is effective for assessing youth with visual impairments.

The analysis of signal duration revealed that the average duration of the exercise for blind individuals was lower than that of sighted individuals, and the agreement between expert evaluations and the re-evaluation by time was moderate. This discrepancy impacts the modeling of automatic classification of the score using supervised learning algorithms. Future work will address this factor to improve the reliability of automated assessments.

Machine learning

Most research that includes balance tests and ML does not consider people with visual impairments and uses other balance tests. Lin et al. (2022), predict the Berg Balance Scale (BBS) score of a participant without professional supervision using Machine Learning regression and IMUs. Bao et al. (2019) analyze the feasibility of using trunk sway data and ML techniques to automatically assess balance, providing accurate assessments at home.

Deep learning

It has been confirmed that by employing DL models it is possible to treat IMU data as a time series, thereby enabling the classification of the punctuation given by a professional for a balance test. Thus, a binary classification has been determined as the most appropriate method to predict whether a participant has good or bad balance when performing Test 3 of the mini-BESTest. A similar procedure was followed by Gil-Martín et al. (2021) in founding the scores for the Y-Test, which was treated as a regression problem. This was in contrast to the desire to obtain the normalized reach distance (NRD) per each leg.

Regarding the number of sensors in the body for this experiment, information from 12 parts of the body was recorded and also analyzed to determine the minimum number of sensors to obtain acceptable performance in the classification of Test 3. The number of sensors was reduced from 12 to six sensors. In contrast, other studies included one, two or eight sensors depending on the activity. It could be because only with one sensor they could obtain one activity in their study, but ours included 14 different activities, and it was the first time a database was created with valuable information that takes into account blind people.

In addition, in this study four types of DL architectures were tested: CNN-1D, LSTM, GRU and CNN-LSTM being the hybrid model, the most accurate in classifying accuracy and F1-score. Kim et al. (2021) used a CNN-ID and GRU ensemble model to assess clinical balance using BBS. They compared it with previous work that used ML instead of DL and obtained 98.4% as the best accuracy of the model. Also, El Marhraoui et al. (2023) utilized a CNN to train and extract weights after training to predict fall events using the Balance Test Score.

Furthermore, in this work, data augmentation was used to improve classification performance and increase data for each class (mini-BESTest score), but the results did not exceed 60%, so it was discarded. As well as us Wei, McElroy & Dey (2020) used this technique to create more variable data from the Center of Mass (CoM) varying the position of x and z, and this with a Balance Evaluation Test to estimate the balance level of each subject if it has or does not fall risk.

Finally, when binary classification was tested using acceleration as input data, it was possible that labels 2 and 1 could be confused in the prediction due to a discrepancy in the evaluation process. To enhance the reliability of the results, it would be advisable to perform further tests with a larger number of data cases, including the evaluation of 0, 1, and 2 (to include a greater number of participants) and to retest the system. This approach could potentially lead to an improvement in system performance.

This study proposes an innovative methodology for classifying the mini-BESTest using IMU data, offering a promising approach to this field. However, it is imperative to acknowledge the limitations of the study. One key challenge is that only two professional evaluators participated in the initial classification, which can introduce bias into the metrics used. Including additional evaluators in future work could refine these metrics and subsequently improve the training and performance of the classification models.

Future work will focus on assessing the specific impact of exercises 3, 11, and 14, where significant differences in balance were observed for blind individuals. In addition, the development of new exercises and evaluation methods will be explored, emphasizing more accessible and widely practiced activities such as pilates, yoga, or taichi (Wang et al., 2021). These alternatives could provide complementary approaches to the mini-BESTest for assessing and improving balance in diverse populations.