Frequency distribution-aware network based on discrete cosine transformation (DCT) for remote sensing image super resolution

Degradation model

Due to hardware device limitations, the RSI resolution cannot meet the practical task’s needs. Two degradation factors are considered: noising and downsampling. Thus, the degradation model can be presented as Eq. (1). (1)${\displaystyle y=Ax+n,}$ where A denotes the downsampling matrix, x and y represent high-quality and low-quality images, respectively. n is the white Gaussian noise. The function of the SR algorithm is to predict high-quality image x through low-quality image y.

DCT

Proposed algorithm mainly uses DCT operation to extract image information in the frequency domain. DCT is a frequency domain transformation method (Khayam, 2003). The two-dimensional DCT operation can be shown as Eq. (2). (2)${\displaystyle A\left(i,j\right)=\alpha \left(i\right)\alpha \left(j\right)\sum _{x=0}^{M-1}\sum _{y=0}^{N-1}I\left(x,y\right)cos\left[\frac{\pi \left(2x+1\right)i}{2M}\right]cos\left[\frac{\pi \left(2y+1\right)j}{2N}\right],}$ for i = 0, 1, 2, …, M − 1 and j = 0, 1, 2, …, N − 1. M is the number of rows and N is the number of columns. The inverse transformation can be expressed as: (3)${\displaystyle I\left(x,y\right)=\sum _{i=0}^{M-1}\sum _{j=0}^{N-1}\alpha \left(i\right)\alpha \left(j\right)A\left(i,j\right)cos\left[\frac{\pi \left(2x+1\right)i}{2M}\right]cos\left[\frac{\pi \left(2y+1\right)j}{2N}\right],}$ and we have: (4)${\displaystyle \alpha \left(i\right)=\left\{\begin{array}{c}\hfill \sqrt{\frac{1}{M}},i=0\hfill \\ \hfill \sqrt{\frac{2}{M}},i\ne 0\hfill \end{array}\right.}$ (5)${\displaystyle \alpha \left(j\right)=\left\{\begin{array}{c}\hfill \sqrt{\frac{1}{N}},i=0\hfill \\ \hfill \sqrt{\frac{2}{N}},i\ne 0\hfill \end{array}\right..}$ By performing Eqs. (3) and (4), we can complete the DCT and IDCT operations.

DCT converts image data from the spatial domain to the frequency domain, which can make the features of the image more prominent in the frequency domain, facilitating further feature extraction and processing. DCT coefficients reflect the frequency energy distribution of the image. The low-frequency signal is distributed in the upper left corner of the image after transformation, and the high-frequency signal is in the lower right corner. DCT is often used for image compression. Here, the DCT operation is used to extract the frequency features of the image. Specifically, we extract the features after the DCT operation, including the correlation information of frequency components between different regions and the non-local information of frequency feature maps at different scales. Therefore, two DCT operations complete the extraction of frequency information in different aspects. The mining of frequency information is helpful for the network to make full use of image features for image reconstruction.

FDANet

The structure of FDANet is shown in Fig. 1. Low-resolution images can be reconstructed into high-resolution images through the network. It is worth noting that this network utilizes a recursive neural network strategy to iterate the basic network multiple times, thereby improving its performance and reducing the parameter quantity. Figure 2 depicts a more detailed introduction to the FA and GF modules.

FA module

In this section, we present the structure of the FA module. The FA module consists of five operations: the DCT operation, rearrange operation, global feature fusion operation, rearrange operation, and IDCT operation. The structure of the FA module is shown in Fig. 3. We first divide the image into 8 × 8 pixel patches. Then, we perform a DCT operation for each patch. DCT can reach the energy weight of the low-frequency signal at the upper left corner of the matrix and the energy weight of the high-frequency signal at the lower right corner of the matrix. Therefore, the distribution of different frequency components in the image can be extracted to complete image reconstruction. The image processing flowchart of the FA module is shown in Fig. 4. In other words, the frequency component of each patch can be divided into 64 classes through the DCT operation. The new 64 feature maps represent the distribution of 64 frequency components from low-to-high frequency, which means the new feature maps are composed of the same frequency components in different regions. This arrangement concentrates the frequency component of the image so that the network can better learn the frequency correlation between different patches. Moreover, a global feature fusion block can fully fuse the feature maps of different frequency components.

The structure of the global fusion block is shown in Fig. 5. It is a row-column decoupling strategy. The size of the input image is b × n × h × w, where b denotes the batch size, n denotes the channel number, h, and w denotes the height and weight of the feature maps. We first perform a convolution operation on the input features. This step will complete information fusion between different feature maps. Then, we transpose the feature maps, and the size of the feature maps is h × n × w. After the convolution operation, the column pixels are fused. Next, we transpose the feature maps, and the size of the feature maps is w × n × h. The convolution operation will complete the feature fusion between row pixels. Finally, we transpose the feature map to the original dimension. The non-local information extraction of the row and column pixels is equivalent to the non-local information extraction of all pixels about the image. This process can extract global information from frequency domain feature maps by performing matrix transpose and convolution operations. It requires less computation than the other global information extraction methods.

The information fusion process is shown in Fig. 6. The central pixel is fused with pixels along the x-axis, then with pixels along the z-axis, and finally with the y-axis. It is equivalent to fusing the central pixel with all pixels in the feature maps. Our row-column decoupling strategy can extract the global information and avoid directly calculating the correlation information between all pixels. The computational complexity of the network is effectively reduced. Finally, the image is rearranged and IDCT operations are performed to obtain the output image.

The FA module rearranges the characteristic information of frequency components so that the network can extract the frequency distribution information more effectively, which facilitates image SR reconstruction.

GF module

The GF module is shown in the second part of Fig. 2. Its basic structure is Unet. Unet is a classical neural network structure that can extract the multi-scale information of the images; it consists of the encoder and decoder. The structure of Unet has four layers and the size of the feature maps are b × n × h × w, $b\times 2n\times \frac{h}{2}\times \frac{w}{2}$, $b\times 4n\times \frac{h}{4}\times \frac{w}{4}$ and $b\times 8n\times \frac{h}{8}\times \frac{w}{8}$, respectively. The feature maps of each layer in the encoder will be transmitted to the corresponding layer of the decoder. This operation can effectively retain the shallow information of the network. We add a frequency information extraction block to the original Unet structure during feature transmission. For example, in the first layer we perform a DCT operation on the feature maps to get the features in the frequency domain. Then, we extract the global features of the image in the frequency domain by global feature fusion block. This global feature fusion block is the same as in the FA module. After the IDCT operation, the features can be transmitted to the decoder.

We perform frequency feature extraction operations on each layer of Unet so that the network can learn the global frequency information of different scales. The feature maps with different scales will contain different frequency components. Therefore, this multi-scale information mining strategy takse advantage of image reconstruction.

Remote sensing dataset1

The algorithms were applied to the UC Merced LandUse dataset. The experiment results are shown in Table 1.

FDANet performs best. HSENet achieves the second-best results, which may be due to the algorithm’s utilization of image multi-scale self-similarity features. We average the algorithm’s results for image reconstruction under different conditions, including multiple scale factors and noise levels. The average PSNR values of IMDN, HSENet, LGCNet, FENet, MAN, and FDANet are 26.89 dB, 27.56 dB, 27.41 dB, 27.55 dB, 27.28 dB, and 27.78 dB, respectively. The average SSIM values of IMDN, HSENet, LGCNet, FENet, MAN, and FDANet are 0.724, 0.744, 0.748, 0.734, 0.749, and 0.775, respectively. Table 1 shows that FDANet outperforms the other competitive methods under various noise levels and scaling factors, which shows the algorithm’s effectiveness.

Remote sensing dataset2

The experimental results on the WHU-RS19 dataset are shown in Table 2. The WHU-RS19 dataset is more complex, thus its average PSNR and SSIM values are lower than the UC Merced Land-Use dataset. The proposed algorithm still performs best. HSENet has the second-best performance. The average PSNR values of IMDN, HSENet, LGCNet, FENet, MAN, and FDANet are 25.97 dB, 26.54 dB, 26.41 dB, 26.53 dB, 26.20 dB, and 26.59 dB, respectively. The average SSIM values of IMDN, HSENet, LGCNet, FENet, MAN, and FDANet are 0.675, 0.702, 0.705, 0.676, 0.695, and 0.731, respectively. Table 2 shows that FDANet performs well in more complex remote sensing datasets, which verifies the algorithm’s robustness.

Visual results

In this section, we present the visual results for algorithm comparison. We randomly selected several scale factors and noise levels to demonstrate the effectiveness of the algorithm in various conditions. Figure 8 is the aircraft scene with a scale factor of 2 and noise level of 0. LR and HR denote low-resolution and high-resolution images, respectively. The low-resolution image is obtained by the nearest neighbor interpolation method. The large image on the left is the whole HR image. We enlarge the area in the red box for algorithm comparison. The large image on the left is the whole HR image. The low-resolution image has severe sawtooth effects. The edge information of the image is lost. Multiple SR algorithms can restore image details to varying degrees. The edges of FDANet are clearer than the other algorithms. HSENet and FDANet have similar performance. The performance of IMDN is limited. Therefore, FDANet performs best.

Figure 9 shows an intersection scene with a scale factor of 3 and noise level of 0. The task of the three times SR is more challenging than two times SR task. LR image will lose more details. SR algorithms can effectively reconstruct image details. Both FDANet and MAN have good image reconstruction results. FDANet can fully mine the image’s frequency and spatial domain information, so it can effectively reconstruct the image details and has the best performance.

Figure 10 shows a baseball field scene with a scale factor of 3 and noise level of 7. This is the most challenging task. The edge of the LR image is no longer clear. All SR algorithms can recover partial edge information. Moreover, FDANet has the best reconstruction results. The edge of the image is sharper, and noise removal is more effective.

In summary, the proposed algorithm has a good visual reconstruction result. The algorithm is robust to different scale factors and different noise levels. Therefore, FDANet can effectively reconstruct image details and edges.

Ablation experiments

In this section, we will perform the ablation experiments. The ablation experiment results are shown in Table 3. PSNR1 and PSNR2 indicate that the experiments are conducted under the UC Merced LandUse dataset and the WHU-RS19 dataset. This article has two innovation modules: the FA and GF modules. Thus, we perform four models for comparison. The backbone of the base model is Unet., which was first proposed in 2015 (Ronneberger, Fischer & Brox, 2015). The base model with the FA module denotes the FDANet model without the gobal frequency feature fusion blocks in the GF module. The GF model denotes the FDANet model without the FA module. The structure of the base model with the FA module is shown in Fig. 11, and the structure of the GF model is shown in Fig. 12. As shown in Fig. 11, the upper half of Fig. 11 is the FA module and the lower half is the base model. The comparison between Fig. 11 and the basic model can demonstrate the effectiveness of the FA module. Blocks of the same color represent Unet feature maps of the same scale. Skip connections were used for feature concatenation between the same scales. The global feature fusion module is specifically introduced in Fig. 6. The global feature fusion module was used during the skip connection process on the basis of Unet, which consists of GF model (Fig. 12). The comparison between Fig. 12 and the basic model can demonstrate the effectiveness of the GF module. Finally, the FA model can extract the similarity characteristics of frequency distribution between different regions and GF model can extract the non-local information of feature maps at different scales in the frequency domain.

Comparing the base model alone and the base model with the FA module proves the validity of the FA module. This frequency-aware strategy is effective. Comparing the base model and the GF module proves the validity of the global frequency feature fusion strategy. This extraction of the global frequency feature is beneficial to image reconstruction. Therefore, the two proposed modules are both effective. FDANet achieves the best performance.