Intelligent lecture recording system based on coordination of face-detection and pedestrian dead reckoning

Proposed auxiliary positioning method

In this section, the auxiliary positioning method based on PDR and particle filtering is explained. A particle filter algorithm has been exploited to fuse rotation information with PDR information. The lecturer’s position is estimated using the PDR step length and heading angle information, and particle filtering is used to fuse the map and camera angle information. This positioning method, based on information fusion and particle filtering, is shown in Fig. 1. This study has been approved by the school of electronic and electrical engineering, Shanghai university of engineering science.

PDR positioning

The aim of PDR (Kemppi et al., 2010) positioning is to obtain the step length, step frequency, and heading of the pedestrian. For this, data are collected using an accelerometer, a gyroscope, and a magnetometer built into a smartphone carried by the pedestrian and used to calculate the pedestrian’s current position based on their position at a previous time. The PDR principle is illustrated using a block diagram in Fig. 2.

The predicted coordinate $\left(E_k,N_k\right)$ can be calculated as follow:

(1) $$\{\begin{array}{c}{E}_k=E_0+\sum _{n=1}^k{d}_n\sin {\theta }_n\\ N_k=N_0+\sum _{n=1}^k{d}_n\cos {\theta }_n\end{array}$$where the initial position is $S_0$, the coordinate is $\left(E_0,N_0\right)$, $d_n$ indicates the step length, and ${\theta }_n$ indicates step heading angle.

Walking direction estimation

The lecturer’s walking direction can be used as the basis for selecting the direction in which to rotate the camera and inferring the lecturer’s direction after they disappear from the camera’s view. The smartphone coordinate system is a relative coordinate system defined by the smartphone screen, as shown in Fig. 3. When the smartphone is placed horizontally, with the screen facing upward, the center of the cell phone screen is the coordinate origin (Zhang et al., 2016). The inertial coordinate system describes the state of motion in the objective world. It coincides with the phone coordinate system and is parallel to the axis of the world coordinate system. In our system, the phone coordinate system is transformed into the inertial coordinate system in a rotation operation. Figure 3 shows a Schematic diagram of the coordinate system (the left is the mobile phone coordinate system, and on the right is the earth coordinate system). Table 1 shows the sensor parameters.

Table 1:

Sensor parameters.

Sensor type	Output data and illustration
Rotation-vector sensor	$value\left[0\right]$, Rotation vector along the x-axis $\left[x\cdot \sin \left({\displaystyle \frac{\theta }{2}}\right)\right]$
	$value\left[1\right]$, Rotation vector along the y-axis $\left[y\cdot \sin \left({\displaystyle \frac{\theta }{2}}\right)\right]$
	$value\left[2\right]$, Rotation vector along the z-axis $\left[z\cdot \sin \left({\displaystyle \frac{\theta }{2}}\right)\right]$
	$value\left[3\right]$, The value of the Rotation vector $\left[\cos \left({\displaystyle \frac{\theta }{2}}\right)\right]$

DOI: 10.7717/peerj-cs.971/table-1

In addition to a gyroscope and an electronic compass that can determine direction, a smartphone also has a virtual rotation vector sensor. This is a software-based sensor that derives data from accelerometer, gyroscope and geomagnetic sensor data.

The output of the smartphone’s built-in rotation vector sensor is a standard quaternions. The rotation matrix around any axis of rotation can be constructed from the basis vectors. The rotation matrix is $R\left(n,\theta \right)$ is calculated as follows:

(3) $$R\left(n,\theta \right)=\left[\begin{array}{ccc}{n}_x^2(1-\cos \theta )& n_x{n}_y(1-\cos \theta )+n_z\sin \theta & n_x{n}_z(1-\cos \theta )-n_y\sin \theta \\ n_x{n}_y(1-\cos \theta )-n_z\sin \theta & n_y^2(1-\cos \theta )+\cos \theta & n_y{n}_z(1-\cos \theta )+n_x\sin \theta \\ n_x{n}_z(1-\cos \theta )+n_y\sin \theta & n_y{n}_z(1-\cos \theta )-n_x\sin \theta & n_z^2(1-\cos \theta )+\cos \theta \end{array}\right]$$

Here, $n$ and $\theta$ represent the rotation matrix, $\lambda$, $q_1$, $q_2$ and $q_3$ are provided by $value\left[3\right]$, $value\left[0\right]$, $value\left[1\right]$ and $value\left[2\right]$, which are calculated using Eq. (4). The parameters of the rotation-vector sensor are shown in Table 1.

(4) $$\{\begin{array}{c}\lambda =\cos (\theta /\phantom{\theta 2}2)\\ q_1=n_x\sin (\theta /\phantom{\theta 2}2)\\ q_2=n_y\sin (\theta /\phantom{\theta 2}2)\\ q_3=n_z\sin (\theta /\phantom{\theta 2}2)\end{array}$$

From this rotation matrix, the attitude of the phone can be solved, the walking direction $\phi$ can be calculated using Eq. (5), and the range of values of the heading angle is $\left[0,2\pi \right]$.

(5) $$\phi =\arctan {\displaystyle \frac{2q_1{q}_2-2q_{0}q{}_3}{1-2q_1-2q_3}}$$

According to the heading angle, it is possible to determine the lecturer’s heading direction. Thus, when the lecturer disappears from the screen, the horizontal servo can determine whether to pan left or right based on the camera, and accurately rotate in the correct direction. Of course, the influence of how the smartphone is carried on the heading angle should be considered. We tested three carrying modes: the hold mode, the coat pocket mode, and the pant pocket mode. For this, a pedestrian carried a smartphone each way as they walked along a specified route. Changes in heading angle radian values were recorded to determine the impact of the carrying mode on the heading angle. As shown in Fig. 4, the different modes resulted in offset angles, but the trend was consistent. Therefore, it was concluded that the heading angle can still be used to estimate walking direction.

Particle filter fusion of map information and camera angle

The particle filter algorithm is performed to fuse landmark point information and PDR information. The landmark point corresponding to the rotation angle of the camera is used as the observation data. We use the number of steps, step length, and direction of the speaker to model the lecturer’s behavior.

Components of proposed system

In this section, all the components of the proposed system are described. There are three modules: (a) face detection module, (b) camera control module, and (c) PDR auxiliary module based on particle filtering. First, face detection is employed to detect the lecturer’s face. Next, the PDR auxiliary module is used to locate the lecturer as they are moving. The lecturer’s location information is transmitted to the server through Wi-Fi. The camera rotation control module has two control strategies: the rotation angle is calculated based on the offset of the picture and the offset of the position information. The framework of the proposed system is shown in Fig. 5.

Face detection module

In recent years, face detection has been widely adopted in computer vision. In the lecturer tracking process, the main function of this module is to detect the face. Considering that our proposed system has edge-based detection, it was deemed suitable to adopt MobileNet. In addition, the Intel OpenVINO toolkit optimizer (Kustikova et al., 2019) has been applied to enhance face detection efficiency.

Backbone

The face detection module is based on the MobileNet-SSD framework. MobileNets (Howard et al., 2017) is an efficient framework that was proposed by Google in 2017. It deploys a low-latency model through two collaterals suitable for mobile edge devices. It is a single-stage detection method that mainly uses depth-separable convolution to decompose the standard convolution kernel. At the same time, two hyperparameters were introduced to reduce the number of parameters and the computational workload. In addition, MobileNet-SSD has two important features. First, it uses MobileNet to replace the VGG-16 background network. Second, the depth convolution in MobileNet separates the standard 3*3 convolution, turning it into 3*3 depth separable convolution and 1*1 point convolution. Each convolution layer is followed by a batch normalization layer and Rectified Linear Units (ReLU) activation function layer; the network implementation is shown in Fig. 6.

OpenVINO

OpenVINO is a toolkit for the rapid development of applications and solutions that Intel developed based on its hardware platforms. The toolkit accelerates and optimizes the legacy APIs of OpenCV, OpenXV vision libraries to run on CPU, GPU, and FPGA. The toolkit is based on the latest generations of artificial neural networks, including CNNs. As shown in Fig. 7, the main components of the toolkit are model-optimizer and an inference engine. The model optimizer is a cross-platform command-line tool that converts trained neural networks from their source frameworks into intermediate representations for inference operations. The inference engine supports the accelerated operation of deep learning models at the hardware instruction set level, as well as the instruction set optimization of the traditional OpenCV image processing library.

We choose the face-detection-adas-0001 network based on the MobileNet-SSD framework. This model is used in driver monitoring and similar scenarios. It has been optimized on the Caffe framework using the OpenVINO toolkit, which can run at 25 frames per second on the CPU. We also compared it with other model frameworks provided by OpenVINO and finally selected it for experimental testing. As shown in Table 2, the deep learning evaluation index Average Precision (AP) was introduced to evaluate these models. It was found that face-detection-adas-0001 had the highest AP.

Table 2:

Comparison of detection accuracy.

Model	Shape for input	Backbone	AP (%)
Face-detection-adas-0001	$\left[1\times 3\times 384\times 672\right]$	Mobilenet	94.1
Face-detection-adas-binary-0001	$\left[1\times 3\times 384\times 672\right]$	Mobilenet	91.9
Face-detection-retail-0001	$\left[1\times 3\times 300\times 300\right]$	SqueezeNet-light+ssd	83

DOI: 10.7717/peerj-cs.971/table-2

Camera control module

Arduino UNO R3 is a crucial component of the proposed camera control module. This module contains the microcontroller ATmega328 and a PT camera. It directs the camera to perform horizontal and tilt movements, extends the monitoring range, and ensures that the lecturer is always within the camera’s field of view. In addition, a reasonable PT control strategy should minimize unnecessary jitter. That is, the PT movement should be smooth. The face detection module acquires the tracking target to guide the movement of the PT camera.

Pan-Tilt camera control strategy

As stated above, we have established the coordinate system related to the PT camera. When the $t^{th}$ frame is set, the target tracking algorithm calculates and outputs the pixel coordinates of the center image of the target frame as $\left(u_t,v_t\right)$, and the target position in the previous frame is $\left(u_{t-1},v_{t-1}\right)$, Fig. 8 shows the camera’s field of view.

We control the rotation of the PT camera by computing the distance between the face and the center of the screen. The face detection module draws the bounding box of the detected face and calculates the offset angle based on the distance between the bounding box and the center. The formula for calculating the offset angle as follow:

(15) $$\left[\begin{array}{c}\mathrm{\Delta }{\theta }_x\\ \mathrm{\Delta }{\theta }_y\end{array}\right]=\left[\begin{array}{c}\arctan {\displaystyle \frac{\mathrm{\Delta }x}{f}}\\ \arctan {\displaystyle \frac{\mathrm{\Delta }y}{f}}\end{array}\right]=\left[\begin{array}{c}\arctan {\displaystyle \frac{\left(u_t-u_0\right)dx}{f}}\\ \arctan {\displaystyle \frac{\left(v_t-v_0\right)dy}{f}}\end{array}\right]$$where $\mathrm{\Delta }{\theta }_x,\mathrm{\Delta }{\theta }_y$ are the offset angle along the x-axis and y-axis, and $dx$ and $dy$ are the physical dimensions of each pixel in the $X$-axis and $Y$-axis, $\left(u_0,v_0\right)$ is the center point of the screen, $f$ is the focal length. The focal length of the camera is 2.8 mm.

Tracking strategy when the target disappears

When the target disappears from the camera’s view, the PDR module uses the coordinates of the lecturer’s position collected by their smartphone to locate the lecturer and calculate the rotation angle. The PT camera is then rotated. As shown in Fig. 9, the offset angle is calculated based on the lecturer’s current position.

The angle of the PT camera is calculated from the position in the real environment, and ${\theta }_2$ is the rotation angle from an initial position, calculated as follows:

(16) $${\theta }_2=\arctan \left({\displaystyle \frac{x_k}{d+y{}_k}}\right)$$

(17) $$\theta =\{\begin{array}{c}{\displaystyle \frac{\pi }{2}+{\theta }_2(0<{\theta }_1<{\displaystyle \frac{\pi }{2})}}\\ {\displaystyle \frac{\pi }{2}-{\theta }_2({\displaystyle \frac{\pi }{2}<{\theta }_1<\pi )}}\end{array}$$where the $\left(x_k,y_k\right)$ is the current position calculated from PDR module, $d$ is the distance between the lecturer’s initial position and the PT camera, and $\theta$ is the rotation angle.

PDR auxiliary module based on particle filtering

When the lecturer’s motion trajectory is smooth, the face detection module can be utilized. Nevertheless, the face detection module will return accurate results when the lecturer moves suddenly or swiftly. This is a universal problem in learning-based tracking methods (Huang, Li & Nevatia, 2013). To overcome this obstacle, we have introduced a PDR-based method for when face detection method is not feasible. At the same time, a particle filter algorithm is applied to further integrate the obtained position information. This can help the system detect the lecturer when they are out of the camera’s field of view and determine their position in the two-dimensional plane. The motion blur and scale-variance caused by rapid movement can likewise be resolved to a certain extent through this operation. The Android mobile application developed in Android studio can obtain PDR information. Through the particle filter deployed on the server, the pedestrian trajectory estimation information is fused with the absolute positioning information provided by the PT camera.

Experimental evaluation

We conducted various experiments to evaluate the performance of the intelligent lecture recording system and to compare it to alternative approaches. Here, the findings are presented, as well as a brief discussion of the impact of the particle filter on positioning error and the advantages and disadvantages of several related systems.

Experimental environment

Experiments in this research are carried out in the classroom, laboratory, and conference room respectively. The computing platform is under Python 3.7 with a 2.8 GHz Intel i7-10750H and 16G RAM. The resolution of the videos is 384*672, and the video is 25 FPS. Figure 10 gives an example of the experimental environment.

The experimental results are presented in the context of two distances used to determine the range of the PT camera’s rotation. As depicted in Fig. 11, the two distances were (1) the distance between the PT camera and the lecturer and (2) the range of the lecturer’s movement. The important distances are shown in Table 3.

In this system, the transmission of PDR data is accomplished by establishing a Socket TCP/IP communication with the lecturer’s smartphone. A laptop is used as the server, and the mobile app is set as the client. Furthermore, serial communication is established between the Arduino controller and the PT camera. The application runs on a Xiaomi 11 smartphone with a sampling frequency of 50 HZ. The smartphone obtains the required data by accessing the built-in API interface of Android Studio. As shown in Fig. 12, the laptop, as the server, communicates with Arduino UNO R3 through the serial port. The smartphone uses sensors to collect data and then transmits the data to the laptop for processing via Wi-Fi to satisfy real-time tracking requirements. Figure 12A shows the hardware connection, including the laptop, PT camera, and control system based on Arduino. Figure 12B shows the mobile app page that displays PDR information. The hardware and software platforms employed in the system are shown in Table 4.

Evaluation metrics

To evaluate the performance of the system, we introduced two evaluation metrics. The first is the $Center\mathrm{\_}rate$. It can be calculated by the Eq. (18).

(18) $$Center\mathrm{\_}rate={\displaystyle \frac{Center\mathrm{\_}num}{Frame\mathrm{\_}num}}$$where $Center\mathrm{\_}num$ is the number of the frames when the lecturer appears in the center of the camera’s field of view. $Frame\mathrm{\_}num$ is the number of all frames in the video. In order to avoid unnecessary shaking, we set a dead zone which is an offset less than 100 pixels from the screen. The camera will not rotate within this zone. It indicates that the lecturer is in the center of the screen. When the lecturer moves frequently, the lecturer is still in the camera frame. Therefore, we considered another ratio $In\mathrm{\_}rate$. It can be calculated by the Eq. (19)

(19) $$In\mathrm{\_}rate={\displaystyle \frac{In\mathrm{\_}num}{Frame\mathrm{\_}num}}$$where $In\mathrm{\_}num$ represents the number of the frames when the lecturer appears in the camera’s field of view.

In addition, to evaluate the system’s robustness, the duration and frequency of invisible speakers in different videos are determined as another dimension of evaluation. The duration is the time that the speaker vanishes from the camera. Here we use slight and severe errors to illustrate the system’s robustness.

1. Slight error: The PT camera performs an incorrect operation, but it recovers immediately within a few seconds (Error time <= 10 s);

2. Severe error: The PT camera performs a wrong move and does not return to the lecturer for a longer time (Error time > 10 s).

Results in different scenarios

The evaluation experiments were conducted in three different scenarios: a conference room, a laboratory, and a classroom. Table 5 lists the percentages of the proposed metrics for the 18 videos recorded in these scenarios. The experiments described in this section were performed with the smartphone in hold mode.

As indicated in Table 5, $Center\mathrm{\_}rate$ of the videos captured in the laboratory varied from 54.80% to 63.89%, and the $In\mathrm{\_}rate$ varied from 84.13% to 88.46%. The average $Center\mathrm{\_}rate$ of the videos recorded in the classroom was 2.44% higher than that of the video recorded in the laboratory, and the average $In\mathrm{\_}rate$ was 3.02% higher. The best performance metrics were obtained in the conference room. The $Center\mathrm{\_}rate$ in the conference room ranged from 64.43% to 67.89%, and the $In\mathrm{\_}rate$ varied from 88.31% to 92.25%. To determine why the performance varied in these different scenarios, we further analyzed the impact of the distance between the camera and the lecturer.

As shown in Fig. 13, where $d$ is the distance between the camera and the Podium, $\theta$ is the camera’s angle in the field of view, $l$ is the horizontal field of view range, and L is the range of the lecturer’s movement. The field of view can be calculated based on the distance and offset angle in this system. Triangle basis knowledge can be used to calculate $l$ as follows:

(20) $$l=2d\tan \theta$$

From Table 6, we know that as d increases, the corresponding field of view will also increase. In these scenarios, a small camera rotation angle allows the camera to cover a large range. Therefore, in the classroom—which had a larger l—we obtained better results than in the laboratory. In addition, the value of d was slightly reduced in the conference room, but the range of the lecturer’s movement L was significantly reduced. In this case, the camera did not need to rotate frequently. At the same time, the lecturer’s walking speed was limited, and there was no sudden movement. Therefore, the scenario in the conference room also returned good results, even though it had the shortest distance between the camera and the podium. In our system, the face detection module requires a minimum head size of 90 × 90 pixels, so d should ideally be 1.6–2.0 m for our system to provide satisfactory results.

Table 7 shows the error counts in three scenarios, the overall runtime of the recordings, the number of error instances, and the total duration of errors. We found that slight and severe errors were rare in our proposed system. The occasional error was caused by the unsmooth rotation of the PT camera.

Results in different carrying mode of the smartphone

We also considered the impact of the smartphone carrying mode on the system’s performance. The smartphone was carried either in the lecturer’s coat pocket or pant pocket, and the video was recorded in the classroom scenario. Table 8 shows the system’s performance when the smartphone was in the coat pocket and in the pant pocket, and these results can be compared with the results of the hold mode presented in Table 5.

The findings indicate that the proposed system’s tracking function is still reasonable when the smartphone is in a coat or pant pocket. The $Center\mathrm{\_}rate$ of the videos recorded with the smartphone in the coat pocket was 58.78–60.25%, and the $In\mathrm{\_}rate$ varied from 84.15% to 86.96%. The quality of the videos recorded when the smartphone was in the pant pocket or coat pocket was similar; however, the heading angle fluctuated to a certain extent due to the shaking of the smartphone in the pant pocket. The metrics were slightly lower for both modes compared to the hold mode. However, good tracking could still be achieved. Therefore, it can be concluded that the smartphone carrying mode has a small impact on the system’s performance and that the hold mode, like a fixed IMU, delivers the best performance. Although it should be noted that, from a cost and complexity perspective, adding a fixed IMU creates additional system costs—there is a cost–performance trade-off.

The impact of the PDR auxiliary module on the system

To evaluate the impact of the PDR auxiliary module, we used only the face detection module to record videos without PDR information. Table 9 shows the metrics for the classroom scenario videos recorded without the PDR auxiliary module. As indicated in Table 9, the performance of the system without the PDR auxiliary module was considerably reduced. The average $Center\mathrm{\_}rate$ was 15.05% lower, and the $In\mathrm{\_}rate$ was about 30% lower.

As shown in Table 10, the count of slight and severe errors in the system without the PDR auxiliary module was significantly higher than in the proposed system, and the error duration time increased considerably. Clearly, the PDR auxiliary module significantly improves the proposed system’s performance.

Particle filter positioning error analysis

Here, we used the average positioning error and the cumulative distribution probability of the positioning error as measurement metrics for positioning performance. The cumulative distribution probability of the positioning error is the cumulative probability that the positioning error is less than a specific value. In this experiment, the lecturer walked on a classroom stage holding a smartphone while the system recorded. In several test cases, the lecturer walked along the same path several times. Both the actual trajectory and the predicted trajectory were recorded. These experimental cases were divided into two groups: one group used PDR positioning without a particle filter, and one group used PDR based on the particle filter algorithm. All positioning errors were counted according to the mean values.

The resulting cumulative probability distribution of the positioning error is displayed in Fig. 14. It can be observed that PDR positioning with the particle filter is significantly better than PDR positioning without it. The positioning error range was greatly reduced by the inclusion of the particle filter algorithm, which improved the system’s positioning performance. At the same time, it was found that the probability of the error range was more than 80% within 2 m, which helps the PT camera to locate the lecturer when they move out of the camera’s view and greatly improves search efficiency. Moreover, because the camera itself has a certain angle of view, we concluded that the system effectively solves some failures associated with locating the lecturer caused by errors in the positioning system.

Comparative analysis of related systems

The proposed system was then compared with existing related systems (Table 11). Multi-camera systems (Taj & Cavallaro, 2011) can be easily deployed for indoor and outdoor localization. However, for such a system to be effective, multiple cameras must collaborate, and the computing power of edge devices is high. Panoramic camera systems (Sun et al., 2020) are simple to deploy, but the associated image distortion needs to be corrected. In addition, although WIFI-PDR Integrated Indoor Positioning’s (Li et al., 2016) is highly accurate, it must collect WIFI offline information in advance. Camera-IR thermal sensors (Tan, Kuo & Liu, 2019) can satisfy the real-time demands with low cost, but it’s not easy to deploy. For instance, IR thermal sensors need to be fixed to the ceiling. When compared with Tan’s system, our proposed system has moderately better experimental results, and our system is portable and easier to deploy. The approach based on Siamese Fully Convolutional Classification and Regression (Guo et al., 2020) performs well in the public target tracking set. Nevertheless, it has significant equipment requirements for high performance, and the effect is not satisfactory in actual lecture-recording scenarios.

In conclusion, our proposed system is a better option for lecture recording than existing systems. Tracking accuracy has been greatly improved due to the introduction of PDR positioning technology with particle filtering. However, our system still has certain shortcomings. For example, it can only perform single-face detection; hence, when there are multiple faces in the picture, it cannot identify the faces. This topic warrants further study in future work.