Mobile augmented reality based indoor map for improving geo-visualization

BLE-based indoor localization

BLE is a wireless personal area network technology designed for applications in the healthcare, fitness, beacons, security, and home entertainment industries. Proximity sensing is one of its applications, which also provides a new method for indoor localization. BLE realizes indoor localization by measure the distance between a mobile device and several beacons. The locations of the beacons are known. Specifically, based on the known location of the beacon, the received signal strength (RSS) of the beacon can be used to estimate the distance from the mobile device to the beacon, which can be expressed as Eq. (1): (1)${\displaystyle RSS\left(\lambda \right)=RSS\left({\lambda }_0\right)-10\eta log\left(\frac{\lambda }{{\lambda }_0}\right)+\mathrm{X}}$

where λ is the distance from the beacon to device, $RSS\left(\lambda \right)$ is RSS of a beacon, λ₀ is reference distance (normally λ₀ equal to 1m), η is the path loss exponent, and X is a zero-mean Gaussian distribution variable with variance ${\sigma }_{\mathrm{X}}^2$.

The localization method is called triangulation. If the location of the mobile device is $\left(x,y,z\right)$, the locations of the beacons are $\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{i}}\right)$, i = 1, 2, …, N. N is usually more than 3. The measured distances are d_i. Then, we can get N equations: ${\mathrm{d}}_{\mathrm{i}}^2={\left(\mathrm{x}-{\mathrm{x}}_{\mathrm{i}}\right)}^2+{\left(\mathrm{y}-{\mathrm{y}}_{\mathrm{i}}\right)}^2+{\left(\mathrm{y}-{\mathrm{y}}_{\mathrm{i}}\right)}^2$. By solving the N equations, we can get the location of the mobile device.

Because of the complexity of indoor environments, the measured distance usually contains error, which makes localization results unreliable. We propose a novel algorithm that improves the localization accuracy of BLE. By the beacon, we can obtain the signal quality index (SQI), which reflects the confidence of estimated distance.

If the SQI is greater than the threshold, we use a triangulation method to calculate the position of the device; otherwise, we use fingerprinting-based method to estimate the location. A weight k-nearest neighbour (WKNN) is adopted for location calculation. The location is calculated by the following equation: p = (w_i∗p_i)(i = 1, 2, …, k),, where p is the estimated location, w_i is the weight, p_i is the location of the ith beacon, k is a parameter, which is determined by experiment. w_i is calculated based on the measured distance between the device and beacon:

1) w_i = 1/d_i,

2) normalize w_i.

PDR

PDR algorithm is utilized to estimate the displacement. If the previous location is $\left(x,y\right)$, the next location is calculated as: (2)${\displaystyle \left(x+l\ast n\ast \mathit{cos}\left(h\right),y+l\ast n\ast sin\left(h\right)\right)}$

where l is the step length, n the step number, and the heading. Step number is obtained by the peak detection algorithm (Mladenov & Mock, 2009). The step detection result is shown in Fig. 3. The step length is estimated using the step frequency-based model (Cho et al., 2010): l = a∗f + b, where f is the step frequency, and (a, b) are the parameters that can be trained offline.

Particle filter-based data fusion

The PF algorithm is based on the sequential Monte Carlo framework, applied to nonlinear and non-Gaussian estimation problems (Arulampalam et al., 2002). A typical particle filter comprises of the following steps:

Initialization: Sampling N particles based on the initial localization result.

Prediction Sampling: Predict a new particle ${\mathrm{p}}^{\mathrm{i}}\left(\mathrm{k}+1\right)$ for each particle $p^i\left(k\right)$ based on the prediction function. In this paper, we use PDR to model the user’s movement for the prediction of multiple particles. The new location ($x_{k+1}^i,y_{k+1}^i$) of ith particle is updated by (3)${\displaystyle \left\{\begin{array}{c}\hfill x_{k+1}^i=x_k^i+\Delta lcos\left(h_k+\Delta h\right)\hfill \\ \hfill y_{k+1}^i=y_k^i+\Delta lsin\left(h_k+\Delta h\right)\hfill \end{array}\right.}$

where Δl and Δh are the distance and heading change obtained by PDR, and h_k is the heading at time k.

Importance Sampling: Calculate weights ${\mathrm{w}}^{\mathrm{i}}\left(\mathrm{k}+1\right)$ for each new particle ${\mathrm{p}}^{\mathrm{i}}\left(\mathrm{k}+1\right)$.

Normalization and Resampling: The weights are normalized and resampled. In the resampling process, particles with low weight are deleted and particles with high weight are duplicated.

In our proposed method, the initial localization is obtained by on BLE-based localization. The particle prediction is realized by PDR algorithm. The weights are calculated based on the distances between the new particles and BLE-based localization results. Based on this online indoor positioning method for AR camera tracking, we fused AR and indoor map for GIS data visualization.

Indoor map pre-processing

A map constructed of planar features will block the real scene; the indoor map is only composed of line feature that looks transparent in AR view. In order to display attribute information on the indoor map, we keep the text information as a texture and map it onto indoor map. Thus, textual information will align with indoor map whenever an application requires the user to zoom in, zoom out or rotate the map.

The user may be distracted by showing entire map of a large space in an AR view. The displayed part of indoor map should be limited within a certain range of the current location. Thus, we divide indoor map by a regular grid and build an index for map data. Each regular grid cell records the minimum bounding box, transverse step, and longitudinal step. During the AR visualization, we get the current position grid as center, and acquired its surrounding eight grids for displaying. This data structure based on a spatial index of regular grid needs low storage and little communication.

AR-GIS visualization

We designed the interface of AR-GIS system that fused the situation and indoor map visualization. As described in Fig. 5, we divided the screen into two parts, which the top one third is displayed the situation information, and the lower two third is rendered the indoor map. Moreover, the indoor positioning result is rendered into the indoor map to show the current user position.

We transformed all these visual elements into the AR coordinate system. As shown in Fig. 6, Fig. 6A is indoor map coordinate system and Fig. 6B is AR coordinate system.

We acquired the rendered part of indoor map according to the current position, and calculated the bounding box of indoor map ABCO and the center point M(x_m, y_m). We put the point M as a new original point, and reduce the size of indoor map in accordance with a certain proportion to keep it within the AR view. Considering the indoor map coordinates in Euclidean space [O; x, y], we define the AR coordinate system as [O; x′, y′]. The coordinate transformation can be described as Eq. (4): (4)${\displaystyle \left\{\begin{array}{c}{\mathrm{x}}^{\prime }=\frac{\left(\mathrm{x}-{\mathrm{x}}_{\mathrm{m}}\right)}{max\left(\left|2{\mathrm{x}}_{\mathrm{m}}\right|,\left|2{\mathrm{y}}_{\mathrm{m}}\right|\right)}\hfill \\ {\mathrm{y}}^{\prime }=\frac{\left(\mathrm{y}-{\mathrm{y}}_{\mathrm{m}}\right)}{max\left(\left|2{\mathrm{x}}_{\mathrm{m}}\right|,\left|2{\mathrm{y}}_{\mathrm{m}}\right|\right)}.\hfill \end{array}\right.}$

Furthermore, we optimized the AR-GIS visualization system from three aspects.

The existing map visualization methods do not consider the direction relationship between map and the real world. As shown in Fig. 7B, the orientation of indoor map always changes with mobile phone, not the real scene. Users commonly identify the north by a compass in the map. However, this approach is not suitable for AR visualization. AR explains the spatial context by overlaying digital data onto the users’ view of the real world. The preferable way to make user understand spatial data clearly in an AR view is not by providing more visual elements like a compass, but rendering virtual data that directly matches the real scene. In order to keep a same orientation between indoor map and the real scene, we first obtained the angle between the mobile phone’s orientation and the north. We obtained the orientation and rotation of mobile phone from multiple sensors. Considering clockwise direction as positive direction, when the phone is tilted by an angle θ away from the north, we rotate the indoor map by the same angle in the opposite direction. The calculation of angle θ in indoor environment needs to account for the deflection angle of the indoor map. As shown in Fig. 7C, the direction of indoor map always matches the real scene whenever the mobile phone rotated, so that users can better understand the map information in AR view.

The foreshortening effects are basic visual rules in the real world. To make the AR visualization more realistic, we rendered the indoor map with foreshortening effects when the phone’s pitch angle changes. We obtain the pitch angle from the sensors in mobile phone, and rotate the indoor map around the X-axis along with the angle change.

As show in Fig. 8A, when the phone is placed horizontally that the pitch angle is approximately equal to 0°, the user can only see the ground through their phone, so there is no foreshortening effects. The pitch angle is getting bigger along with user lifts the mobile phone, the indoor scene appears in the camera view, and AR map shows the effect of “near big far small” as well (Fig. 8B).

We provide the spatial information onto AR view based on the spatial relationship between GIS data and the actual indoor environment. As shown in Fig. 9, O is the current position, V is the camera direction, and θ is the angle to the north. We set two threshold values d₁ and β, which identify distance and the angle of vision. Thus, the quadrangle Op₁p₂p₃ is the field of AR view. When an object appears in the quadrangle, its information will be rendered onto AR view. In addition, if the object fall in the triangle Op₁p₂, its information will rendered on the left of the AR view. Otherwise, the information will be rendered on the right of the AR view.

We build the virtual and real camera alignment by coordinate transformation. For example, a point p in the Op₁p₂, whose screen coordinates of information can be described as: (5)${\displaystyle \left\{\begin{array}{c}x=\frac{\frac{d_2}{2}-d_3}{\frac{d_2}{2}}\ast \frac{width}{2}\hfill \\ y=\frac{d_4}{d_1}\ast height\hfill \end{array}\right.}$

where the width and height are the width and height of screen, d₂ is the distance between p₁ to p₃, d₃ is the perpendicular distance between p to the camera direction v, and the d₄ is the perpendicular distance between p₂ to the perpendicular line segment from p and camera direction v, where a perpendicular line from d₃ intersects the line from p and camera direction v.

If the point p in the Op₂p₃, the screen coordinates of information can be described as: (6)${\displaystyle \left\{\begin{array}{c}x=width-\frac{\frac{d_2}{2}-d_3}{\frac{d_2}{2}}\ast \frac{width}{2}\hfill \\ y=\frac{d_4}{d_1}\ast height\hfill \end{array}\right.}$

Finally, the indoor map and situational information are rendered on to camera AR view by the coordinate transformation.