Prediction of PM_2.5 concentration based on a CNN-LSTM neural network algorithm

Study area

We chose Qingdao, an important city in China, as the research area. Qingdao is located in the south of Shandong Peninsula, east of the Yellow Sea. In Qingdao, the population has grown rapidly in recent years. While many enterprises have flourished, this region is facing challenges related to the rapid development of economic, industrial and urbanization processes. Human factors have led to a sharp increase in the emission of air pollutants, such as PM_2.5, and unique giant sea salt aerosol particles combine with abundant urban aerosols, resulting in a secondary aerosol effect (light pollution problem). Air pollution has become one of the factors restricting sustainable development. To understand the PM_2.5 concentration in Qingdao, a study was conducted based on 20 national ground monitoring stations for PM_2.5. The study area is shown in Fig. 1.

Data sources

Daily historical air quality concentration and meteorological data of Qingdao city from January 1, 2020, to December 31, 2020, were chosen as the research objects and used as input parameters of the CNN-LSTM model proposed in this study. The main air pollutant we chose was PM_2.5. Temperature, pressure, and wind speed were selected as meteorological data indicators. PM_2.5 data were obtained from the Qingdao station data, and the meteorological data used in this study were retrieved from the China Meteorological Data Service Center (http://data.cma.cn/en). Data were interpolated by Kriging interpolation method in ArcGIS software to obtain continuous coverage of meteorological data in the study area, and finally a surface meteorological data set of Qingdao with a resolution of 1km in 2020 was obtained. PM_2.5 imposes a strong random effect in space and time. Thus, we selected three meteorological factors that have the greatest influence on PM_2.5 concentration over Qingdao, namely temperature (TEM), wind speed (IWS) and pressure (PRS). Therefore, the proposed CNN-LSTM model can be evaluated from the data of 20 stations in Qingdao. Table 1 shows the names and information of the monitoring stations.

Data normalization processing and data division

We found that after PM_2.5 data acquisition at the abovementioned sites, the obtained data are usually dirty, with missing values and inconsistent units (Narkhede et al., 2023). Therefore, it is not possible to use these data directly for the model training and relationship mining. To improve the model prediction accuracy and reduce the time needed for the actual training, we carried out pretreatment operations on the PM_2.5 and meteorological data (Zheng et al., 2022). The specific cleaning operations applied to this dataset include modifying the year, month and day information as an index, deleting the data with continuous null values, fill in the missing values with the value of the previous known data. This method is relatively simple and effective, which can maintain the continuity of data and help reduce information loss, thereby improving the accuracy of the model. standardize and normalize the dataset to ensure that it meets the input requirements of the neural network, and averaging the data for every 24 h and reporting the results in hours (Liang et al., 2020). In the model training, due to the different ranges of variable data, it is necessary to normalize the feature data. The normalization formula is shown in Eq. (1): (1)${\displaystyle M=\frac{X-X_{min}}{X_{max}-X_{min}}}$

where M is the value after x normalization, x is the original value of the variable, X_min is the minimum value, and X_max indicates the maximum value. After the original value of each variable is normalized, the dataset is divided into a training set (80%) and a test set (20%) for the input parameters of the model in this study to provide the data utilization efficiency and the model accuracy (Sun & Xu, 2021).

CNN

It was mentioned in the introduction of this article that CNNs are currently used in air pollutant prediction to help extract the spatial information of pollutants. A CNN is a kind of deep neural network that is generally composed of an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer. The convolution and pooling layers are generally applied for feature engineering, and the fully connected layer is used for feature weighting and is equivalent to a “classifier” (Chua, 1997; Chua & Roska, 1993; Girshick, 2015). The network features include “local connectivity” and “weight sharing”, which simplify the complexity of network links, improve the ability of the model to extract abstract features, and alleviate the problems of slow training and easy overfitting of the fully connected network to a certain extent (Bhatt et al., 2021). The convolutional neural network structure is shown in Fig. 2.

LSTM

It was mentioned in the introduction that LSTM has great advantages in PM_2.5 time series prediction. First, recurrent neural networks (RNNs) are a class of recurrent neural networks that use time series as input. The input condition of a recurrent neural network lead to its natural advantage in processing time series, and its parameter sharing structure enables it to better learn the nonlinear features of sequences (Staudemeyer & Morris, 2019; Yu et al., 2019). LSTM is an improved recurrent neural network model that realizes information screening and retention by introducing three “gate” structures: an input gate, an output gate and a forget gate (Qin et al., 2022). Figure 3 shows the internal structure of the LSTM unit, and Fig. 4 illustrate the LSTM network structure. C_t−1 represents the state of the cell at a previous time (_t−1 time), h_t−1 represents the state of the hidden layer of the cell at a previous time (_t−1 time), x_t represents the input at the current time (T-time), C_t is the state of the cell after an update (memory unit), h_t is the state of the hidden layer after an update (memory unit), and HT is the input of the subsequent cell unit(Song et al., 2020). As mentioned earlier, the three gate structures within the cell unit, the forget gate f_t, the input gate i_t, and the output gate O_t, are responsible for updating the cell state and the hidden layer state. The specific functions of the three gates are as follows. The forget gate f _t determines how much the cell state C_t−1 transitions from the previous moment (_t−1 moment) to the current moment, the input gate i_t determines how much input x_t from the current moment (T-moment) contributes to the cell state C_t from the current moment, and the output gate determines how much cell state C_t from the current moment (T-moment) influences the output h_t. Unlike the traditional RNN, which directly outputs h_t−1 from the hidden layer at the previous time as a reflection of the historical state, the LSTM network uses the cell state C_t−1 at the previous time as a reflection of the historical state through the self-circulating memory unit inside the cell and constantly updates the hidden layer state of the network, which has been proven to exhibit better timing processing capability and applicability in practice (Qadeer et al., 2020). The corresponding cell renewal process is shown in Eqs. (2)–(7): (2)${\displaystyle f_t=\sigma \left(W_f\cdot \left[h_{t-1},x_t\right]+b_f\right)}$ (3)${\displaystyle i_t=\sigma \left(W_i\cdot \left[h_{t-1},x_t\right]+b_i\right)}$ (4)${\displaystyle {\overline{C}}_t=tanh\left(W_c\cdot \left[h_{t-1},x_t\right]+b_c\right)}$ (5)${\displaystyle C_t=f_t\times C_{t-1}+i_t\times {\overline{C}}_t}$ (6)${\displaystyle O_t=\sigma \left(W_o\cdot \left[h_{t-1},x_t\right]+b_o\right)}$ (7)${\displaystyle h_t=O_t\times tanh\left(C_t\right)}$

In these formulas, W_f, b_f, W_i, b_i, W_c, b_c, W_o and b_o are the parameters of the LSTM model that are constantly updated during the training process. σ and tanh are the activation functions of the hidden layer of the model, which are responsible for improving the nonlinear representation of the model.

Integrated CNN-LSTM neural network model

In this study, we combined the CNN and LSTM models to address the time series prediction problem and effectively improve the PM_2.5 estimation accuracy. We used the CNN to extract the features of the time series data and designed the LSTM according to the output of the CNN model for prediction (Huang & Kuo, 2018).

The specific structure of the CNN-LSTM model we constructed is shown in Fig. 5. We extracted PM_2.5, temperature, pressure and wind speed information from 20 stations in Qingdao, interpolated the missing values, and normalized the data. The processed data were input into the CNN. In considering the particularity of PM_2.5 time series data, a one-dimensional convolutional layer and pooling layer were designed as the base layer of the hybrid model (Li, Hua & Wu, 2020). In order to better input the output of CNN and LSTM to the fully connected layer, a flat layer is constructed between the LSTM layer and the fully connected layer. A fully connected layer is constructed to decode the LSTM output. Finally, the prediction results of PM_2.5 are obtained by using the proposed model. We used the sigmoid function as the activation function (described in ‘Activation function’) instead of other commonly used activation functions. Because of its special structure, sigmoid function can alleviate the problem of gradient disappearance in neural networks to a certain extent. In addition, an efficient parameter optimizer, the adaptive moment estimation (Adam) optimizer, was used to replace the gradient descent method. In the Adam optimizer, the learning rate of the parameter can be dynamically updated. As a result, the model has more chances to escape local optima (Zhang et al., 2021).

Setting model parameters

The root mean square error (RMSE), mean absolute percentage error (MAPE) and accuracy rate were used as the experimental evaluation indexes of the PM_2.5 prediction model. In the training phase of the neural network model, we first adopt a large learning rate for rapid adjustment to quickly reduce losses. We then use a decaying learning rate to further optimize the training of the network, eventually setting the learning rate to 0.001, the dropout rate to 0.5, the number of iterations to 200, the batch size to 32, and the time step to 4.In addition, by adjusting the experimental parameters and optimizing the number of layers, the optimal model structure parameters were selected, and the number of network layers in the CNN-LSTM model was set to 3 and the number of neurons contained in each hidden layer was set to 16. The specific experimental parameters are shown in Table 2. The Adam optimizer was used as the model optimizer. The L2 regularization term of the matrix composed of all variables in the model was added to the loss function to avoid overfitting (Ryu & Park, 2022).

Activation function

The sigmoid activation function maps real numbers to the interval (0,1) and is mathematically defined as: (8)${\displaystyle f\left(x\right)=\frac{1}{1+e^{-x}}.}$ In neural networks, the sigmoid function is often used as the activation function of the output layer in binary classification problems, and its output can represent the probability that the sample belongs to the positive class. However, the sigmoid function also has some problems, such as the fact that the gradient of the sigmoid function is close to zero when it is far from zero, which may cause the gradient to disappear.

Regularization

Model regularization has always been a core problem in machine learning, mainly with respect to improving the model generalizability. However, most models under the deep learning framework generally have many parameters. Thus, overfitting easily occurs. Therefore, many scholars have proposed regularization strategies suitable for deep learning models, including dropout, which is a universal, computation-friendly but powerful regularization strategy. The basic principle of dropout is that during one iteration of the training model, some hidden layer neurons in the network are randomly discarded. These neurons can be considered as not a part of the network for the time being. However, their weights are retained, and they may be included in the next training iteration. The random exit mechanism of neurons can effectively prevent the interdependence of features.

Evaluation index

To evaluate the effect of the model, the coefficient of determination (R²), RMSE and MAPE were used to compare the accuracy of the model. Among them, the RMSE and MAPE measure the deviation between the predicted value of model and the true value. The lower the values of these indicators are, the better the effect of the model. The R² reflects the degree of deviation between the predicted value of the model and the true value. The higher the R² value is, the lower the degree of deviation, and the better the effect of the model (Liu et al., 2021). The expressions of these three evaluation indicators are as follows: (9)${\displaystyle RMSE=\sqrt{\frac{1}{\mathrm{n}}\sum _{i=1}^n{\left(y_i-{\hat{y}}_i\right)}^2}}$ (10)${\displaystyle MAPE=\sum _{i=1}^n\left|\frac{y_i-{\hat{y}}_i}{y_i}\right|\cdot \frac{100}{n}}$ (11)${\displaystyle R^2=1-\frac{\sum _{i=1}^n{\left(y_i-{\hat{y}}_i\right)}^2}{\sum _{i=1}^n{\left(y_i-\bar{y}\right)}^2}}$ where n is the number of samples, y_i is the i-th actual observation value, ${\hat{y}}_i$ is the corresponding model prediction value, and ${\bar{y}}_i$ is the mean of the actual observation values.

Data visualization analysis

To better understand the distribution characteristics of the PM_2.5 concentration in Qingdao, we present the average PM_2.5 concentration of 20 stations in Qingdao from January to December 2020 through a three-dimensional map. The three-dimensional map of the average PM_2.5 concentration of stations in January is shown in Fig. 6. The three-dimensional map of the average PM_2.5 concentration at the stations from February to December is shown in Fig. S6. Through the longitude and latitude of the 20 stations and the average concentration of PM_2.5, we can clearly see that the concentration of PM_2.5 in Qingdao is distributed in the range 30–70 µg/m³. Due to coal burning in winter, the average PM_2.5 concentration at some stations in Qingdao in January was high, while only a few stations had a low concentration. From this figure, we also note the necessity of predicting and controlling PM_2.5 concentrations in this region.

Results of principal component analysis

We used the principal component analysis (PCA) method to rank the feature importance of factors affecting PM_2.5, laying a foundation for the subsequent input and analysis of the model parameters. In the process of preliminary selection of influencing factors, according to our field investigation and previous studies, compared with other factors, it is found that meteorological factors such as temperature, air pressure and wind speed are the main factors affecting atmospheric PM_2.5 in Qingdao and other coastal cities. Therefore, we ranked the influence of temperature, air pressure and wind speed on PM_2.5, as shown in Fig. 7. We know that the meteorological factors that have the greatest impact on PM_2.5 in Qingdao are pressure, followed by wind speed and temperature. Their proportions are 0.39, 0.38 and 0.23, respectively. Through the dimensionality reduction of the principal components, the original features were transformed into new principal components. We selected these three principal components and calculated the weight of the principal components by using the load coefficient, feature root and contribution rate information. The load coefficient was divided by the square root of the corresponding characteristic root to calculate the linear combination coefficient matrix, and the linear combination coefficient was multiplied by the contribution rate and then accumulated by the cumulative contribution rate to calculate the comprehensive score coefficient. Then, the weights of the comprehensive score coefficient of each factor in the total comprehensive score coefficient were −0.53, 0.62, and 0.58, respectively. The weight coefficients were combined with the original feature data as a new input variables for model prediction. A heatmap of the factor load matrix was plotted to show the strength of the linear relationship between each principal component and the original feature. A heatmap of the factor load matrix, as shown in Fig. 8, was plotted to show the strength of the linear relationship between each principal component and the original feature. From the figure, we note that the correlation of pressure to the original temperature is 0.94.

In summary, we can use the three principal components to capture the pressure, temperature and wind speed data in Qingdao. This information was used to improve the prediction model by calculating the weights of the principal components. This method may help to improve the accuracy and generalizability of the model to better predict the PM_2.5 concentration.

Model performance evaluation results

To compare the performance of different models, we chose two commonly used neural networks, LSTM and the CNN. The prediction performance of different evaluation indicators is shown in Table 3, and the RMSE prediction results of each model are shown in Fig. 9. The MAPE prediction results are shown in Fig. 10. The fitted scatter density plot is shown in Fig. 11. It is not difficult to see that the prediction performance of our CNN-LSTM hybrid model is better than that of a single model. In particular, the predicted value of the CNN-LSTM model proposed in this article is in good agreement with the measured PM_2.5 value in Qingdao. In Table 3, the CNN-LSTM model achieves the lowest RMSE and MAPE values and the highest R² values in the daily forecast of air pollutants. By observing the RMSE and MAPE image curves of the three models in Fig. 9 and Fig. 10, it is evident that the difference between the predicted curve and the actual curve of the CNN-LSTM model is the smallest, followed by the CNN model, and the change trend between the predicted curve and the actual curve of the LSTM model is relatively large. This indicates that the accuracy of the proposed CNN-LSTM model is high, which means that the model can better capture the change and trend of the actual curve during the prediction process and can predict the future value or trend more accurately. The performance indicators of LSTM are as follows: RMSE of 14.367, MAPE of 54.21% and R ² of 83%, while those of the CNN are as follows: RMSE of 11.356, MAPE of 85.36% and R² of 85%. The results show that the prediction accuracy of the deep neural network models including the CNN and CNN-LSTM is better than that of the LSTM model.

In general, from the experimental results, the prediction accuracy of the above two single models is lower than that of the CNN-LSTM model. In contrast, the CNN-LSTM model we constructed fully exploits the advantages of the two models, considers the spatiotemporal diffusion and correlation of the PM_2.5 concentration well, and reduces the prediction error. Therefore, the CNN-LSTM hybrid model has better prediction accuracy compared with that of the neural network.

Analysis of the predicted performance results

In this study, a CNN-LSTM hybrid model based on temporal and spatial correlations was constructed to predict the average daily concentration of PM_2.5 at 20 stations in Qingdao from January 1, 2020, to December 31, 2020. The distribution of PM _2.5 stations in Qingdao is not uniform. In addition, the PM_2.5 concentrations at these 20 stations show a spatial trend of higher concentration in the north and lower concentration in the south, and the seasonal difference is very significant, with a pattern of winter > autumn > spring > summer. In addition, the improvement rate of the prediction effect for PM_2.5 sites in suburban areas is slightly higher than that in urban areas. This is mainly due to the relatively simple terrain and land type of the suburbs, as well as the small population density, so that the PM_2.5 emission sources are less than that of the cities, which is conducive to the establishment of spatial models (Keyu, 2019). In contrast, due to the high population density in urban areas, the interference of PM_2.5 emission sources and the influence of buildings on meteorological conditions, the accuracy of spatial information modeling is affected (Moursi et al., 2022). The predicted performance is shown in Fig. 12. The predicted values are close to the measured values over the whole region. The model has high accuracy, especially at sites with local high values, demonstrating that the CNN-LSTM hybrid algorithm based on temporal and spatial correlations can deal with nonlinear features and time series changes well. In particular, the performance index of the model shows that the model achieves high prediction accuracy and avoids the model overfitting problem, effectively proving that the CNN can effectively extract inherent features and thus improve the LSTM prediction. Therefore, the CNN-LSTM integrated algorithm has a good PM_2.5 prediction effect in Qingdao.