Remote sensing image analysis and prediction based on improved Pix2Pix model for water environment protection of smart cities

Original Pix2Pix model

The original Pix2Pix model is based on conditional general adverse nets (cGAN). Initially, Pix2Pix was used to convert image styles. Figure 2 is a structural diagram of the original Pix2Pix model.

As shown in Fig. 2, the original Pix2Pix model is composed of a generator and a discriminator. The generator is composed of a traditional U-net network. The discriminator is composed of PatchGAN, which divides the input image into N × N, and then calculates the probability of each small block. Finally, the average value of these probabilities is taken as the output of the whole. As a result, the amount of computation is reduced and the training speed and convergence speed can be increased. An input image is received by the generator and converted into a predicted image by encoding and decoding, while the discriminator optimizes the parameters of the generator by discriminating between the predicted image and the real image.

ConvGRU model

One of the problems of the original Pix2Pix model is that time series features of images cannot be extracted, while ConvGRU is suitable for time series feature extraction of images. In ConvGRU, the full connection part of GRU is turned into a convolution operation, which preserves the ability of GRU to extract time series as well as the ability to process image data and extract its spatial–temporal characteristics. So ConvGRU is applied to video detection (Wang, Xie & Song, 2018b), gesture recognition (Zhao et al., 2022) and other fields (Qian et al., 2022).

As shown in Fig. 3, ConvGRU has the advantages of both CNN and GRU, extracting spatial features by CNN and temporal features by GRU, thus extracting temporal features in time series at the same time. ConvGRU can be described as (Shi et al., 2017): (1)${\displaystyle R_t=\sigma \left(X_t\ast W_{xr}+H_{t-1}\ast W_{hr}+b_r\right)}$ (2)${\displaystyle Z_t=\sigma \left(X_t\ast W_{xz}+H_{t-1}\ast W_{hz}+b_z\right)}$ (3)${\displaystyle {\tilde{H}}_t=tanh\left(X_t\ast W_{xh}+\left(R_t\circ H_{t-1}\right)\ast W_{hh}+b_h\right)}$ (4)${\displaystyle H_t=Z_t\circ H_{t-1}+\left(1-Z_t\right)\circ \widetilde{H_t}.}$

Here, * denotes convolution operation, ∘ denotes Hadamard product, σ denotes sigmoid function, W_x∼ and W_h∼ denotes 2-dimensional convolution kernel, X_t denotes input image at the current time, R_t is the reset gate, Z_t is the update gate, H_t is the hidden state at time t, and ${\tilde{H}}_t$ is the candidate set.

U-net model

The generator of the original Pix2Pix model is the U-net network, which was first used to solve the problem of medical image segmentation (Ronneberger, Fischer & Brox, 2015). Figure 4 is the structure diagram of the U-net network.

As shown in Fig. 4, the left side of the U-net is the down-sampling network for feature extraction, and the right side is the up-sampling network for feature fusion. In feature extraction, image information will be lost and image resolution will be reduced. In feature fusion, the feature map with a larger size obtained by up-sampling through deconvolution lacks edge information. Therefore, edge features can be completed through feature stitching to obtain a complete feature map. It is for these reasons that U-net networks are widely used in the segmentation of images (Li et al., 2018; Li et al., 2022; Yao & Jin, 2022) and generation of images (Kim, Yoo & Jung, 2020; Manjooran et al., 2021).

Data scale of remote sensing images unification based on pixel substitution method

In the actual remote sensing image, the scale of each image data is not uniform due to the inversion difference between different images, as shown in Fig. 5.

The red dotted line part in Fig. 5 represents the data scale of remote sensing images. The data scale of each remote sensing image uses the same nine color grades to represent the nine concentration ranges of chlorophyll-a, but the concentration ranges represented by the same color grade on different images are not the same. Therefore, it is necessary to develop a unified data scale for all remote sensing images and replace the pixel values of all remote sensing images according to the unified data scale.

First, the mean value of the concentration range represented by each color grade on all remote sensing images is calculated to obtain the unified data scale, and the calculation equations are shown in Eqs. (5) and (6) below. After that, the Euclidean distance between the concentration range represented by each color grade in the original image and the concentration range represented by each color grade in the unified data scale is calculated as shown in Eq. (7), and then the pixel value of each color grade in the original image is replaced by the pixel value of the color grade of the unified data scale with the smallest Euclidean distance. (5)${\displaystyle L{N}_i\left(max\right)=avg\left(\sum _{n=1}^k{L}_i^n\left(max\right)\right)}$ (6)${\displaystyle L{N}_i\left(min\right)=avg\left(\sum _{n=1}^k{L}_i^n\left(min\right)\right)}$ (7)${\displaystyle \begin{array}{c}\left(L{N}_i\left(max\right),L{N}_i\left(min\right)\right)=\mathrm{arg min}\hfill \\ \left(\sqrt{{\left(L\left(max\right)-L{N}_i\left(max\right)\right)}^2+\left(L\left(min\right)\right)-L{N}_i{\left(min\right)}^2}\right).\hfill \end{array}}$

Here, LN_i(max), LN_i(min) are the maximum and minimum values of chlorophyll-a at grade i after data scale unification, LN_i denotes the concentration of chlorophyll-a at grade i, $L_i^n\left(max\right)$ and $L_i^n\left(min\right)$ are the maximum and minimum values of chlorophyll-a at grade i in the n th sheet in the original data set, k is the total number of samples in the data set, avg denotes the mean value operation, L(max), L(min) refers to the maximum and minimum values of each grade on each remote sensing image.

Remote sensing image repair based on spatial weight matrix

Clouds and abnormal data may cover some areas during remote sensing image sampling, resulting in the lack of data for some remote sensing images, as shown in Fig. 6. Thus, it is necessary to repair the missing area data after unifying the data scale.

The gray area in Fig. 6A represents the missing data caused by cloud cover, the black area in Fig. 6B represents the missing data caused by cloud cover, and the white area in Fig. 6C represents the missing data caused by abnormal data. The repair of the missing area data can be realized by deducing the spatial relationship of the data around the missing area. In describing spatial relationships, some form of distance is usually used as the weight, and the weight is usually the inverse of the distance considering the contribution of the distance to the missing values. As for remote sensing images, the strength of spatial relationships decreases with distance to a stronger extent than linear proportional relationships, so different weight calculations should be used for different spatial relationships.

Therefore, the missing area is repaired by using different weights for different adjacent points based on the spatial weight matrix. The schematic diagram of spatial weight is shown in Fig. 7. When repairing remote sensing images, first determine the contour of the part to be repaired, and then repair from the contour to the middle part until the area is completely repaired.

For the first-order neighbor, the weight is the square of the reciprocal of the Euclidean distance, and for the second-order neighbor, the weight is the reciprocal of the Euclidean distance. Equation (8) shows the specific repair method.

In Fig. 7, the black area is the pixel point with missing data, the dark green is defined as the first-order nearest neighbor, the light green is defined as the 2-order nearest neighbor. (8)${\displaystyle P\left(i,j\right)=\frac{\sum _{a=1}^n{w}_1\times P_1\left(i_a,j_a\right)+\sum _{b=1}^m{w}_2\times P_2\left(i_b,j_b\right)}{n\ast w_1+m\ast w_2}.}$

Here, w₁ is the weight of the first-order nearest neighbor, w₂ is the weight of the second-order nearest neighbor matrix, P(i, j) is the pixel value of the missing site, P₁(i_a, j_a) is the pixel point of the first-order nearest neighbor, P₂(i_b, j_b) is the pixel point of the second-order nearest neighbor, n is the number of first-order neighboring pixels, and m is the number of second-order nearest neighboring pixels.

Time series data filling based on linear interpolation

For the remote sensing image time series collected for a long period of time, the poor remote sensing communication and other phenomena may occasionally occur during the sampling period, resulting in missing remote sensing images at some sampling moments, i.e., unequal sampling intervals, as shown in Fig. 8.

In Fig. 8, the sampling intervals are 2 days, 3 days and 1 day, respectively. In order to supplement the missing image to obtain a complete time series dataset with equal sampling intervals, linear interpolation is used in this article to supplement the remote sensing image time series.

As the original data set has a minimum sampling interval of 1 day, the data set is supplemented according to the sampling interval of 1 day. The equation is as follows: (9)${\displaystyle P\left(m+\lambda \right)=P\left(m\right)+\frac{\lambda \left(P\left(n\right)-P\left(m\right)\right)}{\left(n-m\right)}.}$

Here, P(m + λ) represents the pixel value of the day m + λ, λ is the number of days from m,m< λ <n, m , n are the dates, and P(m), P(n) are the pixel values of the two dates already in the original data. The schematic diagram of linear interpolation is shown in Fig. 9.

As can be seen from Fig. 9, the pixel values of any one of the two days can be obtained by the pixel values of the two days that are not adjacent to each other.

Remote sensing image prediction method based on 3D-GAN model

To address the problems with existing pixel-level prediction methods, namely that it is difficult to fully extract the time series features from an image, the model is prone to gradient explosion and overfitting, the following improvements are made for the original Pix2Pix model:

1) In the generator, the original Pix2Pix model only uses the U-net network, which can not predict the time series of images. In this article, ConvGRU is connected in series with the U-net network to extract features from image time series. On the one hand, the channel attention mechanism is added to the ConvGRU , and the channel attention mechanism convolutional gated recurrent unit network (CAM-ConvGRU) is constructed, which enables the network to learn important information independently and improve its ability of the network to extract image time series features; on the other hand, the residual structure is added to the down-sampling process of U-net network to avoid gradient explosion or over-fitting, which can realize feature extraction and pixel-level prediction of remote sensing image time series.

2) In the discriminator, the original Pix2Pix model inputs the input image into the discriminator as a label, without considering the time correlation. In this article, the image of the historical moment does the data on the channel to do the overlay as the label, and the time series information is incorporated in the criteria to provide the distinction between real and fake images. Figure 10 is a structural diagram of 3D-GAN model.

As shown in Fig. 10, CAM-ConvGRU obtains the remote sensing image of the 31st future moment by extracting time series features from the input 30 historical moments images and then inputting them into the Unet network incorporating the residual structure for coding and decoding operations, and the remote sensing image of the future moment is input to the discriminator with the labels obtained by superimposing the input 30 historical moments on the channel, and the parameters of the generator and discriminator are optimized through adversarial training to finally obtain the accurate prediction image.

Spatial and temporal distribution prediction of water environment based on remote sensing images

After getting the remote sensing image of the future moment, the chlorophyll-a concentration at each pixel position in the remote sensing image is displayed visually, and according to the magnitude of chlorophyll-a concentration, it is possible to make a judgment on the water environment pollution situation at each pixel position, thus realizing the prediction of the spatial and temporal distribution of the water environment pollution. For the locations with serious water environment pollution in the future moment, corresponding measures can be taken in advance to prevent the occurrence of water environment pollution.

Results of unified data scale based on pixel substitution method

A unified data scale was determined based on Eqs. (5) and (6), and the unified data scale is shown in Fig. 17. According to the Fig. 17, the pixel value for each color grade in the original remote sensing image is replaced with the pixel value for the corresponding color grade in the unified data scale according to Eq. (7), thus completing the pixel replacement of the unified data scale is completed. A comparison of remote sensing images before and after unifying the data scale is shown in Fig. 18.

As shown in Fig. 18, the left figure is the original remote sensing image, and the right figure is the remote sensing image after the unified data scale. It can be seen that the color changes of pixels in some areas of the remote sensing image before and after the unification. By unifying the data scale, the same color grade in all remote sensing images represents the same chlorophyll-a concentration range.

Results of remote sensing image repair based on spatial weight matrix

According to Eq. (8) for remote sensing image repair, the results are shown in Fig. 19, which shows a comparison of remote sensing images before and after repair.

As shown in Fig. 19, black and gray areas indicate data loss due to cloud cover, and white areas indicate data loss due to data abnormality. As shown in Fig. 19, the missing part of the data is supplemented by the surrounding pixels. First, determine the outline of the missing part, then fill the outline of the missing part with the above spatial weight matrix calculation rules, and then fill the middle part gradually until the missing part is completely filled.

Results of time series data filling based on linear interpolation

Figure 20 shows the time series of remote sensing images after linear interpolation obtained using Eq. (9).

Figure 20 depicts the real data in the original data set as March 13 and March 16, and the filling data obtained by linear interpolation as March 14 and March 15. By linear interpolation, the sampling time interval in the data set is set to 1 day.

Parameter setting

A discussion of the 3D-GAN model parameters is presented in this section.

a) Generator parameter settings

The 3D-GAN generator is composed of CAM-ConvGRU and U-net network in series. The parameter settings of CAM-ConvGRU are shown in Table 1. Parameters in Table 1 were derived based on the empirical method and the data format required. The U-net network is divided into the up-sampling network and the down-sampling network. The parameters of the down-sampling network are shown in Table 2.

An empirical method was used to determine the number and size of convolution kernels in Table 2. There are seven layers of up-sampling network in U-net, and the parameters are shown in Table 3.

As shown in Table 3, the number of deconvolution kernels and the kernelsize of the deconvolution are also empirically derived. Step sizes for upsampling and downsampling should be the same.

b) Discriminator parameter setting

The discriminator constructed in this article is the PatchGAN model, and the network parameters are shown in Table 4.

As shown in Table 4, patchGAN has five network layers. This network uses the remote sensing image of chlorophyll-a concentration at 30 times in history to predict the remote sensing image of chlorophyll-a concentration at 31 times. First, 30 historical remote sensing images of chlorophyll-a concentration are superimposed as video frames, and an Adam optimizer is used to optimize the network with a learning rate of 0.001.

Prediction results of remote sensing images

The test set in this article predicts remote sensing images from November 16, 2011 to December 30, 2011, a period of 45 days. Due to space limitations, only some representative prediction images are displayed (November 22, November 20, December 9, December 20, December 24, and December 28. Figure 21 shows the prediction images for 3D-GAN, which represents the improved Pix2Pix model presented in this article.

Figures 21A and 21E represent the real image and the predicted image based on the 3D-GAN model. As can be seen intuitively, the predicted image using the method described in this article is quite similar to the real image.

It is proposed in this article to evaluate the prediction quality of the model more objectively by evaluating structural similarity (SSIM), peak signal-to-noise ratio (PSNR), cosine similarity (cosine), and mutual information. SSIM measures image similarity based on brightness, contrast, and structure; PSNR can reflect the mean square error between two images; cosin can determine the similarity between two images by calculating the cosine distance between the vectors; mutual information describes similarity by calculating the mutual information between two images. In Table 5, the average and maximum values are shown for the similarity between the predicted image and the real image for each prediction model. The more similar the predicted image is to the real image, the closer the predicted result is to the real result. The range of chlorophyll-a concentration in each part can be clearly seen through the predicted image.

As can be seen from the last row of Table 5, the prediction results of 3D-GAN proposed in this article maintain good consistency with the real images in terms of brightness, contrast, structure, mean square error, cosine distance, and mutual information.

Prediction results of spatial and temporal distribution of water environment based on remote sensing image

Due to space constraints, Fig. 22 shows only the actual and predicted images of the remotely sensed images acquired on November 22 and December 20.

Taking Fig. 22 as an example, in the overall view of the predicted 45 days, the chlorophyll-a concentration range is higher in the left area of the center of Taihu Lake as well as the left edge part of Taihu Lake, so it is more likely to have water environment pollution, but the chlorophyll-a concentration is generally lower in the right area of the lake. For example, in the predicted image on November 22, the chlorophyll-a concentration was significantly higher in the upper left part of the lake than in other areas; and in the predicted image on December 20, the chlorophyll-a concentration was higher in the left area of the center of Taihu Lake and the upper edge area, while the chlorophyll-a concentration was at a lower level in most of the rest of the area. Therefore, by strengthening the management of the locations with obviously high chlorophyll-a concentration, we can prevent the emergence of water environment pollution and reduce the harm caused by water environment pollution.