Representation of negative numbers: point estimation tasks using multi-reference sonification mappings

Theoretical background

Apparatus

The Mobile Auditory Graph (MAG) interface was built on a 9.7-inch Samsung Galaxy Tab S2 screen with the Android 7 operating system. A mobile graph application with multimodal input was designed by enabling mobile touch screen gesture interaction for the auditory graph (see Fig. 1).

The prototype has four conditions:

1. Pitch-only, where the point to be estimated is represented by one single note representing its position on the Y-axis

2. Single reference, which is similar to condition 1 but each note is preceded by a reminder of the pitch corresponding to Y = 0.

3. An approach where notes are played representing each 5th of the distance from 0 to Y-max, up to the value just below the point to be estimated, and then a final note representing the actual point itself. For example, if the notes can range in the scale from 0 to 100, to represent a point to be estimated at 65, points would be played representing the following values: 0, 20, 40, 60, 65.

4. Multiple references with fixed step size, similar to Metatla et al.’s (2016) work. In this condition, notes will be played from 0, in fixed steps sizes of 10, up to the value just below the point on the Y-axis to be estimated, and then the note itself. For example, the representation of 65 would include notes for Y = 0, 10, 20, 30, 40, 50, 60 and 65. We set Y-max to 100, One single instrument will be used for all displays rather than the two that were used in the previous study.

A pause of 500 ms was used between all the notes representing points in condition 2, 3 and 4, in all sonification conditions. The purpose of these studies is to test the effectiveness of the prototype with the help of sonified graphs in two types of multimodal interaction: playback and swipe.

Experimental design

Statistical analysis

We used the unbiased root-mean-square error (RMSE) to measure the point estimation error. The RMSE was calculated by taking the differences between values (sample or population values) predicted by a model (standardized estimated or forecasted values = f) and the values observed (true values = o) as shown in the Barnston’s formula (Barnston, 1992): (2)${\displaystyle {\mathrm{RMSE}}_{fo}={\left[\sum _{i-1}^N\frac{{\left(z_{f_i}-z_{f_0}\right)}^2}{N}\right]}^{1/2}.}$

The statistical results provided in the experiment include the mean, standard deviation (SD), median, and interquartile range (IQR) from point estimation tasks of participants. The mean is calculated as the sum of the errors between the true value and the recorded value divided by the number of observations. The standard deviation is a measure of the amount of variation or dispersion of a set of values. The median is the middle value separating the higher half from the lower half of the data sample. The interquartile range (IQR) is the difference between the first quartile (25th percentile) and the third quartile (75th percentile), representing the middle 50% of the data. This explanation clarifies the methodology for readers.

Further statistical analysis was also performed to find whether any significant differences existed across all conditions of the tasks. Additionally, we considered using covariance to further explore correlations between study results. Covariance measures how much two random variables vary together. By analyzing the covariance between the conditions, we can gain insights into the degree to which changes in one variable are associated with changes in another variable.

Ethical considerations

The research protects the information of BVI participants in accordance with ethical standards. We ensured confidentiality and ethical handling of participant data throughout the study. Participants were informed about the purpose of the research, and their consent was obtained before participation. All data collected were anonymized to protect participants’ privacy.

Study 1: BVI participants’ point estimation with representation of negative numbers

The results were then calculated across all subjects by calculating the RMSE between the estimated values and the true values. For further analysis, the results of the mean, standard deviation (SD), median, and interquartile range (IQR) from 20 BVI participants are calculated as shown in Table 2.

These calculations aimed to determine if there are relationships between the performance of point estimation tasks and the condition used to perform the tasks. After calculating the RMSEs, the values were plotted in four boxplots and mean plots to visualise the distribution of the error for each method and each type of graph.

As seen from the boxplots from Fig. 2, the multi-reference mode used in conditions 3 and 4 shows better performances in the point estimation tasks, indicated by smaller distributions as compared with those using pitch-only mode and zero with single reference modalities. We performed a non-parametric significance test with a confidence level α = 0.05 to validate our hypothesis for all conditions. Our null hypothesis is that the RMSE mean of all the conditions are equal. A Kruskal & Wallis (1952) test was used to check the RMSE mean between the conditions. The test does not assume normality in the data and is much less sensitive to outliers. The calculated P-value less than 0.05 led to the conclusion that there are significant differences between the conditions (p = 1.09 × 10⁻³).

Post-hoc analysis with the Wilcoxon test was performed to determine which levels of the independent variable differ from each other. Adjustments to the P-values were calculated to control the familywise error rate (FER) or to control the false discovery rate (FDR).

The pairwise comparison in Table 3 shows that only the pairs conditions 1 vs. condition 4 (with P-value 0.0061) and condition 2 vs condition 4 (with P-value 0.0064) are significantly different (p < 0.05).

To analyse the performance of BVI participants on sign polarity estimates, their respective values across all BVI participants for all conditions were calculated. We employed a stimuli detection task containing a mix between positive and negative numbers over the 40 trials. The set of positive and negative scores was divided equally between each of the 20 trials in random positions for all four conditions.

We calculated the percentage of correct polarity sign estimates. For example, if all 20 participants correctly predicted negative polarities during a trial of negative numbers, then the percentage is 100. However, if two participants falsely estimated the number as positive, then the percentage dropped to 90. The same rules apply to positive trials.

Separation between four conditions was conducted to determine the relationships of the performance of negative number representation tasks and the condition. Our null hypothesis is that the polarity sign between conditions is equal. The result indicated no significant differences among the conditions (Kruskal–Wallis chi-squared = 7.5951, p = 0.052).

Study 2: Sighted participants’ point estimation with representation of negative number

The results for all subjects were then estimated by evaluating the RMSE between the estimated values and the true values. For further analysis, the results of the mean, SD, median, and interquartile range (IQR) from 20 sighted participants are calculated as shown in Table 4.

These calculations were made to assess whether there are relationships related to the execution of point estimation tasks and the condition. The results were plotted in four boxplots and mean plots to visualise the error distribution.

As seen from the boxplots in Fig. 3, the multi-reference mode used in conditions 3 and 4 shows better performances in the point estimation tasks, indicated by smaller RMSEs are represented with lower median and smaller distributions as compared with those using pitch-only mode and zero with single reference modalities. A Kruskal–Wallis test also identified significant differences among the conditions for sighted users (p < 0.001), confirming that multi-reference conditions generally provided more accurate point estimation than other tested methods.

As the Kruskal & Wallis (1952) test is significant, a post-hoc analysis using the Wilcoxon test was performed to determine which levels of the independent variable differ from each other. The pairwise comparison in Table 5 shows that only the pairs condition 1 vs condition 2 (with P-value 0.82) that are not significantly different (p < 0.05).

The representation of negative numbers was calculated across all sighted participants between conditions to determine whether negative numbers could be represented in terms of their components in the auditory modality using the same scheme with BVI participants.

Our null hypothesis is that the polarity sign between conditions is equal. There were no differences between the conditions (Kruskal–Wallis chi-squared = 3.942, p = 0.27).

BVI participants vs. sighted participants result

A combined analysis of both groups across all conditions (80 data points) was conducted to determine if there were correlations in performance between the BVI and sighted users (see Table 6).

As seen from the boxplots in Fig. 4, the BVI participants have better performances in the point estimation tasks in term of lower errors represented with their lower mean and median as compared with sighted participants. It is also interesting that the SD and IQR of the BVI participants were higher, indicating that the point estimation errors are more spread out. As a result, the estimation of the sighted participants has a lower SD, indicating that their error is more clustered around the mean value. A non-parametric significance test with a confidence level of α = 0.05 was conducted to validate differences between the users. Two samples of data remain independent if they originate from different populations and the samples do not influence each other. We can use the Mann–Whitney–Wilcoxon test (1947) to determine if the population distributions are identical while not suspecting that they conform to the normal distribution. The differences between sighted and BVI participants in the point estimation task are significant (W = 25600, p < 0.001).

Analysis of the point-estimation tasks

Our first research question on the point estimation performance task across four conditions was directed to address our hypotheses H1, H2, and H3.

In general, the BVI participants produced more accurate results or fewer errors in conditions 3 and 4 which used the multi-reference sonification mappings. This was indicated by its respective lower mean, median and narrower distribution compared with conditions 1 and 2 as shown in Fig. 2. Nevertheless, further post-hoc analysis revealed that only condition 1 vs condition 4 (p = 0.006) and condition 2 vs condition 4 (p = 0.0064) differed significantly. The other pairs were not found to be significantly different (p > 0.05) (see Table 3).

The results for sighted participants showed even greater differences when compared with BVI participants. The accuracy of the multi-reference modes at condition 3 (Median = 16.98) and condition 4 (Median = 10.15) were much smaller, being almost less than or equal half of the RMSE means for condition 1 (Median = 30.01) and condition 2 (Median = 30.28) (see Table 4). Their respective boxplots also showed narrower distributions, as depicted in Fig. 3. This finding was supported by the results of pairwise comparisons which revealed that all pairs were significantly different except between condition 1 vs condition 2 (p = 0.82) (see Table 5).

Therefore, as predicted in hypotheses one and two, both VI and sighted participants produced higher point estimation errors when using sonification mappings with fewer reference tones (conditions 1 and 2) compared to the multi-reference mappings of 20 steps and 10 steps. These results are in line with Metatla et al.’s (2016) findings who had similar results using narrower scales than the one used in our experiment. It suggested that the multi-reference approaches provide more “anchor” points or more guidance to assist in the point estimation tasks.

Furthermore, H5 predicted that BVI participants would have better performance on point estimation tasks than the sighted group. As displayed in Fig. 4, the BVI participants performed more accurately on point estimation tasks, represented by a lower mean and median of RMSE values when compared with sighted participants. The result of the Mann–Whitney Test (1947) statistics also showed that the differences were statistically significant (W = 25600, p < 0.001).

However, when we compared data for each condition separately (see Table 7), it seems that the RMSE values of conditions 3 and 4 for both groups of participants were not different. Meanwhile, in conditions 1 and 2, sighted participants made twice the number of errors than BVI participants. This shows that BVI participants performed more accurately on point estimation tasks with fewer references (i.e., conditions 1 and 2).

When the reference was added for every multiple of 20 or 10, both groups showed somewhat equal performance (${\tilde{x}}_{\text{cond.3}}=15.02$, ${\tilde{x}}_{\text{cond.4}}=10.24$ for VI; and ${\tilde{x}}_{\text{cond.3}}=16.98$, ${\tilde{x}}_{\text{cond.4}}=10.15$ for sighted participants, respectively). Therefore, both BVI and sighted groups had similar performance in the multi-reference conditions, with both groups performing rather better in condition 4 (step10 sonification mapping) than condition 3 (step20 sonification mapping). These results tend to show that all participants seemed to perform better as the number of reference points increased, in this case when the range from 0 to Y_Estimate was split into 10ths rather than 5ths.

Interestingly, by comparing the SD and IQR of each condition, we observed that the data variability of sighted participants was far lower than that of BVI participants across all conditions.

The participants’ demographic characteristics revealed that several BVI participants a had better level of musical training than the other BVI participants. Our analysis of the data revealed that the more varied data and outliers came from BVI participants with less musical training. Meanwhile, our sighted participants generally had no better musical levels than BVI participants, whose musical level distribution was more varied. Further studies are needed to investigate the possible relationship of the level of musicianship for effective point estimation in this context.

We felt the use of the log/exponential mapping made a noticeable difference to the accuracy of point estimates in this study, compared to the accuracy of point estimates in our preliminary studies (Putra, Sutarto & Anasanti, 2021). Although the overall results did not change substantially, this mapping solution effectively addressed usability issues that several participants had previously reported.

We also found no issue in this study as that happens in different timbral streams issue in our preliminary studies. In this issue, participants mistakenly guessed the coin as piano and vice versa, resulting for example, 70 turning to 20 and vice versa. In other words, whether Y_Estimate is greater than or equal half of Y_Max (Y_Estimate ≥ 0.5 * Y_Max), it could mistakenly perceive as below half of Y_Max (Y_Estimate < 0.5 * Y_Max) and vice versa. While it rarely happens in study 3, the solution to use only one type of sound minimizes this potential issue.

Analysis of negative number reference

Regarding negative number reference, our hypothesis is stated as follow:

H4: & Users will show better polarity sign selections when using the single point and single reference sonification mappings as compared with the one-reference and the multi-references mappings of 20 steps and 10 steps.

The results of the polarity sign task revealed no significant difference across all conditions in both in the BVI participants (Kruskal–Wallis chi-squared = 7.5951, p = 0.051) and sighted group (Kruskal–Wallis chi-squared = 3.942, P-value = 0.267). Table 8 summarizes the descriptive statistics of the percentage of correct polarity signs. Despite the conditions used, both BVI and sighted participants demonstrated high performance in accuracy with all mean values and medians being above 90%.

Metatla et al. (2016), used a polarity-based approach using an exponential function—instead of linear mapping—from the smallest negative number to the largest positive number. In our research, however, we attempted to validate that negative numbers are represented mentally in the form of component representation values, not a holistic representation. This is our main reason to implement the component representation approach for negative numbers by using the same positive mapping reference for the digit—as described in Table 1—and adding a sign before the digit with a “sonar” sound.

Unfortunately: we cannot compare our results with those of Metatla et al. (2016) because he did not describe the results of the polarity data in his paper. However, it is clear in relation to a weakness in Metatla et al. (2016), that the task of point estimation becomes inefficient as the numbers to be represented get further from 0.