Technology-assisted white cane: evaluation and future directions

Literature search, inclusion/exclusion criteria

For collecting the relevant literature, published in the last eight years (2010–17), the authors followed Moher et al. (2009). The authors searched and browsed Google Scholar, Springer, ScienceDirect, ACM digital library, and IEEE. We tried different search queries including “white cane”, “virtual white cane”, “technology-assisted white cane”, “assistive tools for visually impaired and handicapped”, “blind assistive technologies”, and “navigation aids”, and kept the publication year as 2010–17. In the search results against each query, each entry was scrutinized to check whether the query terms appear in the title or snippet and, if yes, was opened in new tab to read in detail its title, abstract, and references (if available). If deemed relevant, the full-text of the article was downloaded to apply our inclusion/exclusion and eligibility criteria. Also, the authors performed the manual inspection of the references list of each selected full-text publication to identify relevant and related papers. This includes research papers that were published beyond the specified range but are still mass produced or commercially available. We adopted the PRISMA checklist, adapted from Moher et al. (2009) and depicted in Fig. 1, at different stages of our exploration, from search to the final selection of full-text articles to include in the literature review. Our eligibility and inclusion criteria consider research publications that:

• enhance the utility of white cane with the existing technologies either for indoor, outdoor or both types of operating environments

• are published beyond the specified range but their developed ETAs are still mass produced or commercially available

• place the sensor unit on the white cane

• discuss virtual white cane, devices that mimic the attributes of the traditional white cane to assist blind users.

The exclusion criteria include articles that:

• are unable to assist blind users through technology

• discuss assistive devices, which are unable to mimic or extend white cane for navigation

• have less academic significance or with unauthorized publishers or repetitive ideas and thus lack in novelty or originality.

The inclusion/exclusion criteria resulted in 36 articles. Also, there are two systems whose publications are unavailable, and therefore, their details are included from their official websites. This results, in total, 38 systems, which are included for comparison and evaluation using the proposed evaluation framework and in the identification of research trends and future research opportunities.

Evaluation criteria and framework

Figure 2 presents the nine criteria that we have identified from the selected 38 ETAs. These include (i) operating environments; (ii) use of sensors (single, multiple); (iii) positioning of sensory unit on white cane; (iv) sensor covering range and area; (v) mode of operation; (vi) computational devices; (vii) functionality; (viii) localization techniques; and (ix) user-system interaction (input/output). Table 1 summarizes each of these criteria with their purpose, possible values, and representative symbols, which are then used in Table 2 (’Summary of Key Observations’) to evaluate and compare the state-of-the-art technology-assisted white cane solutions along with the results and research implications.

Operating environments

Technology-assisted white cane has been designed from its usage perspective either for the indoor environment—enclosed areas including small and large-buildings (Koyuncu & Yang, 2010), outdoor environment—spaces outside these buildings including streets, playgrounds, roads, etc., (Hersh & Johnson, 2008) or both (Table 2, column 3). Figure 3A depicts that most of the existing systems are for both indoor and outdoor environments. It means that the trend is to design and develop a generalized technology-assisted white cane, which can adapt itself to the changing environment and keep the users away from changing/choosing apps that treat each environment differently. This adaptation is required to reduce the cost, training, and cognitive load on the blind users as they would not have to buy, use and become used to with several devices.

The use of sensors

Several different types of sensors have been used to assist blind people in navigation. Their selection depends upon the nature of the problem and the corresponding operating environment. In the selected studies, either a single sensor was used or multiple sensors were fused together (see Table 2, column 4). Wang & Kuchenbecker (2012) use only ultrasonic sensor (US) to detect objects coming in front of the users. Pyun et al. (2013) use three ultrasonic and one infrared sensor to detect the extent of an obstacle. Figures 3B and 3C divide the use of sensors into sensor type and single vs. multiple, respectively. Most of the studies used multiple sensors in different combinations including, e.g., US with IR, US with GPS, US with Cm, US with IR and Water Sensor. The most frequently standalone sensors include Cm and US. Sometimes a single type of sensor is used in different combinations, e.g., to sense in multiple directions. However, the trend is towards using single or multiple types of sensors in detecting obstacles in front, left, right, and head level of the blind user. It means that the system should assist the blind users in detecting obstacles not only at the front but also at the trunk level. In addition, each sensor is exploited for a different purpose: US detects obstacles in front of the user, infrared for inclinations, stairs and distance estimation, camera for obstacle recognition, laser for range estimation, a color sensor for color sensing which is used to differentiate the navigation path from the surrounding environment , tilt angle for detecting the slope angle and RFID tags for identifying essential landmarks. Among these sensor, US is widely used (29 out of 38 systems), followed by camera and then infrared, while the remaining ones are used only in specific situations, e.g., GPS for localization, water sensor for detecting water on the ground or puddles, gyroscope for orientation, accelerometer for step detection, etc. The studies encourage the use of multiple types of sensors in combination, which may potentially increase the perception of blind users regarding their surroundings. However, the relationship of the selection of sensors with the three primary techniques associated with a white cane is not that simple. For example, a user can quickly benefit from their contact with the ground (shorelining) and the interpretation and classification of sounds (echolocation) through the white cane tapping. The situation becomes different when it comes to obstacle detection, where not only the identification of the type of object is an issue but also their position and distance from the blind user. Therefore, the selection of the type and number of sensors should be made in a manner that has no effects on the first two techniques but improves the ability of obstacle detection and extends the range beyond the physical length of the cane. Only a few studies (Bouhamed, Kallel & Masmoudi, 2013; Nada et al., 2015; Okayasu, 2010; Pyun et al., 2013) attempted to supplement the shorelining technique only. The design of the virtual white cane needs attention, as the nonavailability of the physical cane affects the echolocation and shorelining capabilities of the users, although they do support obstacle detection due to their inherent design for detecting obstacles.

Positioning sensors on white cane

Sensors can be mounted at different positions on the white cane. The positioning of the sensors affects the performance of the whole system as well as the use of a white cane. Nada et al. (2015) place one ultrasonic sensor near the bottom of the cane for detecting obstacles in front of the user and another near the handle to detect obstacles at head level facing upward. A water sensor is also placed at the bottom of the cane to detect water and puddles on the ground. Mutiara, Hapsari & Rijalul (2016) place one sensor at cane handle for detecting trunk-level obstacles and the other at the cane bottom with proper angular placement for detecting holes and hitches.

Virtual white cane is a blind assistive aid designed with the possible characteristics of traditional white cane by using a range of sensors, to collect surrounding data, and a computational unit for data analysis, processing, and presentation to the blind user (Vera, Zenteno & Salas, 2014). Croce et al. (2014) use a smartphone camera for navigation by reading and analyzing the color patterns drawn on the floor using taped colored lines. The detected color lines are drawn on the screen and identified by the user while sweeping their finger on the screen. The smartphone generates vibrations, when the finger touches the line on the screen, telling the user that they are going in the right direction.

Table 2, column 5 (Fig. 3E) contributes the evaluation of the chosen systems based on the criterion of where the sensor unit is on the white cane. The trend is either supplementing the white cane by mounting multiple sensors or adapting handheld devices as a virtual white cane. For obtaining the maximum sensing capability, three familiar places on white cane have been considered, which include the cane handle, center, or bottom. Most of the studies use the cane handle (upper part) (21 of 38, 55.3%). However, for a more reliable solution, the weight, shape, security, assembly, and folding of the white cane should be considered while placing the sensor unit on the white cane. If possible, the sensing unit should act as a plug-and-play unit, which can be easily attached to the white cane when required and removed otherwise (Hersh, 2018).

Sensors sensing range and coverage

The type of navigational environment impacts the selection of sensors, their sensing capability (precision), power consumption, portability, usage, range and coverage with specified angular placement (Ercoli, Marchionni & Scalise, 2013). A blind user finds it difficult to detect obstacles that lie beyond the range of white cane. The sensing range of ETAs should not exceed a certain level to avoid detecting irrelevant or unnecessary objects (Mehta, Alim & Kumar, 2017). The cane-mounted or hand-held sensors sense the front (head- and ground-levels) and measure the covered area in degrees. Saaid, Mohammad & Ali (2016) carried out experiments to identify a suitable position for the angular placement of sensory unit on a white cane. They placed sensors on the upper and bottom portions of the cane respectively (90° , 45° ) and found that the upper position of the white cane with 90° is the best place for positioning sensors.

Table 2, column 6 summarizes the sensing range (in meters (m)) of the sensors which can be divided into three sub-criteria including front range (FR), upwards range (UR), and angular coverage area with angular placement (ACA/AP). The sensing range is 0.1 m to 5 m for FR and 0.2 m to 4 m for UR, however, few studies mentioned the value for ACA only and others for AP only (see column 6 in Table 2), e.g., FR and UR sensors are used based on user needs, sensor capabilities, and the nature of the target environment. The intended use of these sensors is to warn the user about the obstacles incoming in their way, and therefore, the sensing range can be made limited to a specified range to avoid detecting foreign objects. Also, some of the systems in Table 2 come up with mode changing capability, e.g., Miniguide, which enables the user to switch mode for working in the indoor/outdoor environment by adjusting its sensing range.

Operation mode

The operation mode (Table 2, column 7) specifies whether the assistive tool is a standalone, cane mounted, or backpack processing unit. Considering the specific needs of blind users, the targeted operating environment, and the functionalities, researchers exploited several operation modes. These include standalone handheld, cane-mounted, cane-mounted with a hand-held device or cane-mounted with back-pack computation unit. Table 2 (Fig. 3F) shows that the majority of the existing solutions (24 out of 38, 63.2%) use a white cane with the mounted sensory unit without requiring any additional device. Only a few considered handheld device (about 28.9%) or and keeping the computational unit as backpack device on the back of the blind user.

Computational devices

The computational devices (Table 2, column 8) process the sensed data to make it usable for navigation. These devices can be smartphones, computer, microcontrollers and analog circuits. The selection of a specific type of computational device depends upon speed, memory, Analog-to-Digital (AD) conversion, and power consumption. Figure 3G shows that the majority of cases used a microcontroller (29 out of 38, 76.3%), followed by the computer as a backpack (3 out of 38, 7.9%) and smartphone (5 out of 38, 13.2%). Multiple computational devices were also used, e.g., in AlAbri et al. (2014).

Functionality

An ETA has one of the four possible uses including obstacle detection, its avoidance, navigation, and localization. Researchers have exploited these functionalities in several combinations, e.g., obstacle avoidance with localization. The desired functionality is offered by exploiting technologies in a multitude of ways including, e.g., tags for localization, smartphones for obstacle detection and navigation. Several solutions offer additional functionalities, e.g., Al-Fahoum, Al-Hmoud & Al-Fraihat (2013) recognize the material of obstacles such as wood, plastic, steel, and glass. Ye, Hong & Qian (2014) recognize the obstacles along with estimating cane pose using Visual Range Odometry (VRO) method. Batterman et al. (2017) connected a tactile button to the handle of the white cane for accessing iOS navigation apps such as Blind Square and Seeing Eye, where the input chosen by the blind users is spoken loudly by the VoiceOver app to assist them in making correct selections. The benefit of adopting this approach is accessing the smartphone without holding it. Rizvi, Asif & Ashfaq (2017) designed a wearable glove to detect obstacles at user front and by using GPS communicates the location of the blind user in the form of text messages.

Table 2, column 9 (Fig. 3D) shows that the majority of these systems (23 out of 38, 60.5%) detect obstacles at the user’s front, left, right, or head level but are unable to offer additional details (hidden semantics) regarding the surrounding objects. The trend is toward mitigating the issue of obstacle avoidance. From column 9, it can be understood that obstacle detection is among the core functionalities and basic techniques associated with a white cane, which have been given significant attention as well as priority in the design of ETAs. In almost 97% of the studies, the leading research design is around the obstacle detection. About 18% support obstacle recognition, whereas echolocation and shorelining attracted few studies. This is in line with our discussion on the selection of sensors in ‘The Use of Sensors’ that users are comfortable with echolocation and shorelining and that obstacle detection should be focused. Among the studies that implicitly consider echolocation and shorelining are presented in Table 2, where E+ stands when a given study affects echolocation (enhances or degrades it) and E− otherwise. On the other side, S+ means that the designed ETA has a positive impact on shorelining and S− otherwise. However, targeting obstacle avoidance alone will not resolve the issue of assisting blind users in navigation as more contextual information including, e.g., the size of the object, its shape, and type is required to build a mental model of the surroundings. Only a few studies focused on obstacle recognition, localization, echolocation and shorelining.

Localization technology

Localization is used to determine the current user position and direction (Hersh & Johnson, 2008). Several localization technologies are used. These include QR-Codes, GPS, and RFID. Table 3 summarizes technologies used for the localization of blind and visually impaired people and reports on their operating environment (indoor, outdoor, or both), accuracy, sensing range in meters (m), data transmission rate in megabytes per second (Mbps), carrier frequency (in hertz (Hz), kilohertz (KHz), megahertz (MHz), and gigahertz (GHz)), infrastructure required, strengths and limitations. Table 2, column 10 shows that only five studies used GPS, one used RFID tags, and two used QR-Codes.

User-system interaction

A blind user interacts with an assistive solution in several ways. Regarding this interaction, the input can be push-button/switches, keypad, and joysticks. The sensed data is processed and presented in several output formats including auditory, vibration, or their combination. Table 2, column 11 (Fig. 3H) shows that most of the studies do not specify their data input methods (16 out of 38, 41.4%) in which 12 used push-buttons/switches, three used keypad, and only one joystick). The majority of the systems use auditory with vibration as the output (19 of 38, 50%), followed by vibration (13 of 38, 34.2%), and then auditory (5 of 38, 13.2%). The audio output is complete but gets affected by external sources and questions the privacy of the user (O’Brien et al., 2014). This can be handled if earbuds or cochlear implants are used, but they should not block the echolocation and other essential sounds from the surroundings that aid in understanding the context and help in localization and safe navigation. Vibration needs direct physical contact with the human body for better results and is not affected by external sources including noise (as in audio output) as well as the privacy concerns (O’Brien et al., 2014).

An ETA that replaces the white cane (and has no other means of tapping the surface) affects the capabilities of blind users associated with the use of echolocation significantly. Examples of such devices include virtual white canes, which are hand-held devices with no physical contact with the ground to collect information. In addition, ETAs that produce continuous auditory output (or vibration patterns) may negatively affect the echolocation-based capabilities i.e., obstacle detection and localization of blind users associated with the use of echolocation technique. This is due to the fact that, either the user will be unable to hear other echoes or their attention span gets affected, respectively. Therefore, it is essential to consider the importance of echolocation while designing ETAs so that the blind user may exploit the resulting solution to the fullest. In a similar way, no direct physical contact with the ground, walkways, pavements, etc., makes it difficult for a virtual white cane to detect such surface textures and communicate to the blind user in an informed way. Therefore, researchers need to focus on this issue so that a more navigation-supportive solution could be designed. In this regard, the use of image processing in detecting and capturing different surface textures and analyzing them could be an excellent research opportunity, if considered.

Other factors

In addition to the nine criteria mentioned in the previous subsections, other factors and aspects need discussion. These aspects include cost, power consumption, weight, size, shape, ergonomics and the social acceptance of the designed ETAs. These aspects are briefly discussed in this Section.

The concept of “cost” is many-faceted in that it can be defined by several things including the price of sensors, their accuracy, power consumption, the number of sensors to be employed, processing unit/computational devices, and the possible effect of these on the weight, shape, and size of the white cane. Table 3 points out several important details of these sensors, which together with these factors may potentially define the cost of the reported systems. For example, in Table 2, the cost of each of these systems can be established by looking at column 3, 4, and 8, i.e., the operating environment, use of sensors and the computational devices, which have a direct impact on the price, shape, size, weight, and a power consumption of the resulting white cane solution.

As already discussed, a solution that is designed for both indoor and outdoor environments will be cost-effective as users do not have to use/buy multiple devices to deal with each environment separately. Regarding the effect of sensors on the cost of the white cane, solutions that use less number of low-priced but efficient sensors (real-time processing with low overhead on the battery) will be considered as cost-effective. For example, the US, IR, and inertial sensors provide average accuracy with less power-consumption (3–5 v), and small size can be efficiently and cost-effectively used with a white cane. In contrast, RFID tags and readers are expensive in the case when they are used in large number to cover more area (technically not feasible especially for outdoor environments). Although GPS and Camera sensors give pinpoint accuracy (especially Camera), they are more power-hungry, involve comparatively greater processing overhead, and are expensive than the others. Therefore, the selection of type and number of sensors should be made carefully so that the targeted system operates equally efficiently in indoor/outdoor environments, multiple weather conditions, with low overhead of processing, power, price, and ergonomics. Above these, in the case of commercially available solutions, their price also makes a difference regarding cost. For example, SmartCane (52$) is much cheaper than UltraCane ($1,295) and MiniGuide (545$) and offer almost the same functionalities.¹ Finally, a computational device that is light, smart in size, shape, and weight, and energy-saving is preferred to reduce the cost of the targeted solution further. For instance, SmartCane and Miniguide use a microcontroller as a computation device and therefore, cost-effective regarding low-processing overhead, a minimal number of US sensors, and low power consumption.

The battery life is an significant criterion for ETAs, especially in outdoor environments when the user is out for a long time. However, it can even be problematic in indoor environments in regions where power-outage occurs frequently. Therefore, solutions that are power-hungry become a natural rejection for the users in such circumstances. A solution can be power-hungry if it uses comparatively more sensors than the others and it involves greater processing overhead regarding the number of sensors, type of sensors, computational devices, functionality, user-system interaction methods, etc. In Table 2, solutions that use GPS and Camera are more power-hungry than US and IR. Also, solutions that use multiple sensors than a single or two would be more power-consuming, see, e.g., Niitsu et al. (2012); Alshbatat & Ilah (2013). Moreover, solutions that use a laptop as the computational device will hardly run for 3–4 h, more tiring than a microcontroller, which runs for many hours. For instance, Kinetic Cane and a co-robotic cane are more power hungry because of the number and type of sensors as well as the computational unit than UltraCane, which is less power-hungry.

The weight, size, and shape are other essential yet related factors that need consideration especially when it comes to the use of a white cane. These aspects affect the natural use of a white cane and can be ergonomically and cognitively stressful for a blind user if not handled with care. For instance, using a computational device directly on the white cane or the daily usage device which does not label the user as visually impaired will be much efficient and blind-friendly in contrast to the back-pack computational devices or other bulky hardware even if it has no effects on the weight, size, and shape of the white cane. Also, these factors have a direct effect on the social life of a blind user, e.g., wearing bulky hardware with a white cane may give strange looks to the blind user with low social acceptance especially for children playing in the surroundings. Therefore, a solution should be comfortable for the blind user in terms of size, shape, weight, as well as its social acceptance in the society so that the blind user feel confident while navigating in the surroundings.

In Table 2, Kinect Cane is larger and heavier than Miniguide and Palm Sonar as the former uses the laptop as computational backpack device than the latter ones that use a microcontroller in the form of a handheld or pluggable unit on the white cane. Regarding the shape, Kinect Cane looks more awkward and difficult to manage than Miniguide as the former uses the camera, Microsoft Kinect sensor and accelerometer sensors and laptop computer as computation device than the latter, which uses ultrasonic sensors and microcontroller for computation. It is, therefore, necessary that the size, shape, and weight of the resulting ETA should not affect the utility and salient features of the white cane, which include folding, water resistance, ease in maintenance and conversion (back) to the white cane (when desired). Also, the ETA should be shockproof. These features could be better achieved in ETA by exploiting the handle of the white cane as compared to other areas in traditional white cane.

The last but not the least factor is the issues of ergonomics and social acceptance of the ETAs. The introduction of bulky hardware to be positioned on the white cane or human, in either case the user will quickly get tired and refuse to use it. The ETA should minimize the learning curve and help the user to quickly adopt it for several activities with minimal information and cognitive overload. The user-system interaction should be designed carefully so that it may not affect the walking speed of the user; if possible, the hands should be free (for in case holding/carrying items such as food), with full attention for the recognition and avoidance of dangerous obstacles (Hersh, 2018). The output can be vibration, haptic or auditory. However, the continuous auditory output may potentially overburden the user with the chances of being interfered with the elimination of natural cues from the surrounding such as traffic noise and echoes from the white cane. This can be handled if earbuds or cochlear implants are used, but they should not block the echoes and other essential sounds from the surroundings that aid in understanding the context and help in localization and safe navigation. Also, the ETA should not affect echolocation, shore-lining and other vital clues that are possible when the white cane is used alone. The vibration and haptic feedback could be a better alternative provided that the optimal number of patterns is used so that the user is not overburdened with information and cognitive overload. According to Lopes et al. (2012), the ETAs should ensure the safety of the user during navigation especially crossing the road, passing through heavy traffic, and making visible the user to others using a blinking LED light (when the sensors sense the dark). In addition, for extended outdoor trips, they proposed that the device could be charged using photovoltaic solar cells. However, these aspects should be considered with care so that these can be socially accepted and may not be an unnecessary hazard for the blind and visually impaired people. Overall, the ETAs should be able “to create a rough mental picture of the surrounding environment, where relevant information is efficiently transmitted without affecting the ergonomics, safety, and social acceptance of blind and visually impaired users, so that they can navigate freely, independently, and confidently in the society.