A crowd clustering prediction and captioning technique for public health emergencies

Computation of density maps for crowds

The density map of crowds can better and intuitively explain the profile of the crowd. The density map of crowds is drawn by labeling the pedestrian head, replacing the kernel with the geometric adaptive Gaussian to predict the head size of the pedestrian and converting it into density maps, which are then input into the network for training. Assume that the coordinate position of the head center from a pedestrian in an image locates at a point x_i, the crowd image with the centers of N pedestrian’s heads in the picture is represented as follows: (1)${\displaystyle H\left(x\right)=\sum _{i=1}^N\delta \left(x-x_i\right).}$

The geometric adaptive Gaussian kernel function $G_{{\sigma }_i}\left(x\right)$ is used to calculate the density map F, which can be present in the following formula: (2)${\displaystyle F\left(x\right)=\sum _{i=1}^N\delta \left(x-x_i\right)\times G_{{\sigma }_i}\left(x\right)}$

where ${\sigma }_i=\beta \overline{d^i}$ and the average distance x_i from the m heads of the nearest pedestrian are $d^i=\frac{1}{m}{\sum }_{j=1}^m{d}_j^i$. The simulation results show that the effect is the best at β = 0.3.

Improved multi-scales, CNN

The main idea of multi-scale CNN is to fuse the data obtained from the three networks and then extract features to achieve better results. Considering the perspective, the network sets up three kinds of networks: large, medium and small. The improved convolution kernels are: the size of L column convolution kernels are $\left[\left(11\times 11\right),\left(9\times 9\right),\left(7\times 7\right)\right]$, M column convolution kernels are $\left[\left(9\times 9\right),\left(7\times 7\right),\left(5\times 5\right)\right]$ and S column convolution kernels are $\left[\left(7\times 7\right),\left(5\times 5\right),\left(3\times 3\right)\right]$. After the images are fed into the three networks, the output data are merged for the density map. The structure diagram of the network is shown in Fig. 2.

The loss function adopted by MCNN network is shown as follows: (3)${\displaystyle L\left(\Phi \right)=\frac{1}{2N}\sum _{i=1}^N{∥F\left(X_i,\Phi \right)-F_i∥}_2^2}$

where Φ denotes the parameters that can be continuously trained in MCNN, N represents the counting of entire images in the training phase, X_i represents the i − the input, F_i indicates the actual density map extracted from the i − the image, $F\left(X_i,\Phi \right)$ is the corresponding density map and L represents the loss function between the calculated density map and the ground truth.

The following two general evaluation values are selected to evaluate the performance for choosing the training model: MAE and MSE. The calculation formula for the two evaluation values is as follows: (4)${\displaystyle MAE=\frac{1}{N}\sum _{i=1}^N\left|z_i-{\tilde{z}}_i\right|}$ (5)${\displaystyle MSE=\sqrt{\frac{1}{N}\sum _{i=1}^N{\left(z_i-{\tilde{z}}_i\right)}^2}.}$

Crowd movement status

The above content introduces the network used in this article and the improvement part in detail. The purpose of using this network structure is as follows: (1) generating crowd density map by network prediction. (2) The head coordinate points of individual pedestrians in each surveillance video frame are predicted through the network. (3) The pedestrian coordinate points are extracted to calculate the characteristic values of the crowd motion state.

The crowd behavior is in a state of constant motion. In this article, the dynamic eigenvalue changes are used to determine whether abnormal crowd sudden dispersion behavior occurs, which are defined as the following three crowd movement state eigenvalues. The first is the average kinetic energy of the crowds. In the monitoring scenario, the average kinetic energy of the crowds will increase when sudden crowd dispersions occur. The second is the crowd density value. Due to the limited monitoring range under the surveillance video, when the crowd breaks out, the crowd will leave the surveillance video and the crowd density value will decrease accordingly. The third is the crowd distribution entropy. The crowd distribution in the monitoring area will change when crowd dispersion occurs. As the crowd dispersion proceeds, the crowd distribution entropy will become larger and more prominent with the crowd dispersion.

(1) Average kinetic energy of the crowd.

Kinetic energy is simply the energy of an object due to its motion. The traditional method determines the feature points by extracting the corner information of the foreground object and then calculates the crowd kinetic energy according to the motion vector of the identified feature points. In this article, the kinetic energy refers to the average kinetic energy from crowds in monitoring areas, and the position $\left(x_0,y_0\right)$ of the center point of the image is selected for each frame of the image.

As the reference point, the average distance s_i Between the network predicted coordinate points in the ith frame and the reference point is calculated. Assume that j pedestrian heads coordinate points are anticipated in the i − the frame. Then, the average distance s_i+1 of the (i + 1)-th frame is calculated and the speed of the first frame is calculated according to the frame rate. Therefore, we can calculate the average kinetic energy of crowds. The formula is presented as follows: (6)${\displaystyle s_i=\frac{1}{N}\sum _{i=1}^N\sqrt{{\left(x_j-x_0\right)}^2+{\left(y_j-y_0\right)}^2}}$ (7)${\displaystyle E_{avg}=\frac{1}{2}m{\left(\frac{s_{i+1}-s_i}{t}\right)}^2}$

where E_avg Represents the average kinetic energy of crowds, t is the time of a single frame and s_i is the average distance of the ith frame. Since the size of moving objects in the motion scene is similar, so m = 1.

(2) Crowd density value.

The crowd density value is used to judge the proportion of the corresponding area occupied by the crowd in a frame. The improved MCNN can generate the density map and simultaneously calculate the predicted number of people in the frame. The specific crowd density value is shown as follows: (8)${\displaystyle D_i=\frac{\lambda N_i}{S_i}}$

where we regard the number of people predicted in the ith frame as N_i, λ denotes the correction factor of the counting of people in an image and S_i Indicates the image area in the ith frame. We set S_i = 1 to simplify the computation.

(3) Crowd distribution entropy.

Entropy is used to calculate and estimate the amount of information, reflecting random time’s uncertainty. In this article, the idea of information entropy is used to calculate the distribution of the crowd. If the crowd is scattered, the entropy of the crowd distribution will increase, and vice versa. Firstly, the coordinates are normalized to make them between $\left[-1,1\right]$. Then, the area is divided into 20 continuous small regions on average, which are $\left[\left(-1,-0.9\right),\dots ,\left(0.9,1\right)\right]$. The crowd distribution entropy is calculated using the following formula: (9)${\displaystyle S\left(n\right)=-\sum _{i=1}^k{p}_{i}log2p_i}$ (10)${\displaystyle p_i=\frac{\text{count}\left(y_k\right)}{\sum _{i=1}^k\text{count}\left(y_k\right)}.}$

Scene captioning model

According to the models above, we can achieve the scene about a happening event in the environment of health emergence. To accomplish the news text about this event, we propose a double-LSTM to generate captions for the scene. The model is presented in Fig. 3.

Given the scene information I, we first feed it into the word generation attention (WGA) to attend to the previous information h_t−1. Before computing attention, we must apply the word x_t−1 and the cell features A to achieve the h_t−1. Then, we feed h_t−1 and the results of attention $h_t^a$ into the language LSTM to obtain the predicted word y_t. Therefore, we can achieve a whole caption $\left\{y_0,y_1,\dots ,y_t\right\}$ for the current scene to get the news text. The entire process can be formulated as follows: (11)${\displaystyle h_{t-1}={\text{LSTM}}^a\left[A,x_{t-1}\right]}$ (12)${\displaystyle h_{t-1}^l={\text{LSTM}}^l\left[h_{t-1},h_t^a\right]}$ (13)${\displaystyle y_t=\text{Softmax}\left(h_{t-1}^{l}W+b\right)}$

Dataset and implement details

This experiment is carried out on an ordinary PC with AMD Ryzen 5 3600 CPU, 3.60ghz, 16G memory, RTX 2070 SUPER graphics card, 8G. The improved MCNN network is built in Anaconda3 + Pytorch1.3.1 + cuda10.1 + cudnn7.5.1 + Python3.7.1 environment for training and testing and the final crowd state eigenvalue analysis is completed in Python3.8 environment.

All population estimation experiments are performed on the ShanghaiTech dataset. There are 1,198 labeled images in this dataset, which includes two different parts which are Part_A and Part_B. In Part_B, the images are sparser than those in Part_A. ShanghaiTech dataset is first established in Zhang, Zhou & Chen (2016), where they donate 300 images to train the model in Part_A and 182 images are regarded as the samples to test. Part_B includes 400 training images and 316 testing images. We use the videos of abnormal crowd activity located indoors in the dataset UMN of the University of Minnesota to predict the trend of crowd aggregation. In addition, we apply the MSCOCO to evaluate the scene captioning model for news text collection.

The results and comparison

To evaluate our method, we compare our method with MCNN, MSCNN, CMTL, Switching CNN, SaCNN, TDF-CNN and CSNet models. We present the experimental results in Table 1.

We can conclude that compared with the MCNN model in the ShanghaiTech dataset, the MAE and MSE of our model decrease by 22.0 and 35.7 on Part_A and decrease by around 16.5 and 39.7 on Part_B, respectively. Compared with CMTL, MAE and MSE on Part_A decrease 26.2 and 14.7 and MAE and MSE on Part_B decrease 9.8 and 17.8, respectively. In addition, compared with MSCNN, Switching CNN, SaCNN and TDF-CNN, our method has an absolute leading position on Part_A and comprehensively outperforms the above techniques on Part_B. However, when comparing with TDF-CNN, our approach lags slightly behind on Part_A and beats TDF-CNN on Part_B, which indicates that our method possesses wider affection than TDF-CNN in the scenario while testing Less foot traffic. In summary, compared with other competitors, our approach can better predict the crowd gathering in different scenes. It can fully provide the necessary predictive information for news text collection and social problem prevention.

Experimental analysis of crowd aggregation

The eigenvalue calculation in this article is based on the video of indoor abnormal crowd activity in the UMN dataset of the University of Minnesota. The detection and analysis of two weird videos, random burst and same-direction burst, are carried out, respectively. The crowd’s density and the crowd’s distribution entropy, we distinguish the irregular and the same direction of the weird videos.

According to the comparison in Figs. 4, 5 and 6, at about 170th frame, the three eigenvalues all begin to change significantly, the kinetic energy and crowd distribution entropy begin to increase sharply and the crowd density gradually decreases. According to the corresponding frames in the video, it can find that the inflection points of the change of the eigenvalues of the three states corresponded to the moment when the crowd started to disperse abruptly in the video. At this moment, the crowd suddenly disperses and starts to run and the kinetic energy and distribution entropy gradually increase. The crowd density gradually decreases as pedestrians run out of the monitoring area. We can find from the figure that the average kinetic energy of the first increase then decreases slowly until about 245th frame returns to the initial values of kinetic energy. The main reason for this anomaly which is different from the former, is pedestrians do not entirely run out of the monitor screen, keep steady and stop running slowly back to walking posture after the 245th frame of the video. Because the pedestrians in the surveillance video are fewer, it can be seen that the value of distribution entropy also starts to level off at around 250th frame.

Results of scene captioning

To evaluate our scene captioning model, we choose the LSTM, SCST, UpDown and UpDown+RD to compare with our method. As shown in Fig. 7, where B@N, M, R, C and S denote the evaluation metrics of BLEU-N, METEOR, Cider and Spice, respectively, our model can achieve the best performance while comparing with other models, especially in the term of Cider scores (C). In the term of BLEU-1 (B@1), we can find that our model surpasses others just a little. However, our model can obtain the obvious advantages, which means that our model can be fed with the scene information and generate accurate text information for the happening event, which can be helpful for the news text collection.