Reconstruction of super-resolution from high-resolution remote sensing images based on convolutional neural networks

Nonlocal module

Convolution operations typically process a local neighborhood one at a time, which can lead to limitations in capturing global dependencies. To overcome this, some algorithms have been developed based on nonlocal mean filtering (Buades, Coll & Morel, 2005), which searches for similar pixels globally across the entire LR image. These methods calculate weighted scores for all pixels in the image, allowing distant pixels to simultaneously contribute to a specific position’s response.

The research proposes integrating a region-level nonlocal module (RL-NL) before and after running the SRNet, as depicted in Fig. 4. The nonlocal operations within this module can gather relevant features from the entire image. Mathematically, the nonlocal operations can be expressed by Eq. (11). (11)${\displaystyle y_i=\frac{1}{\sum _{\forall j}f\left(x_i,x_j\right)}\sum _{\forall j}f\left(x_i,x_j\right)g\left(x_j\right)}$ where “i” denotes the output feature position, “j” represents all possible positions, and “x” and “y” designate inputs and outputs of the nonlocal operations, respectively. The pairwise function, f(x_i, x _j), calculates the relationship between x _i and x_j, while the function g(x_i) calculates the representation at position j.

To evaluate the relationship between the two positions, an embedded Gaussian function is employed that calculates the similarity in the embedded space, as expressed in Eq. (12): (12)${\displaystyle f\left(x_i,x_j\right)=exp\left({\left(W_{\theta }{x}_i\right)}^T{W}_{\varphi }{x}_j\right)}$ where W denotes the weight matrix. Considering a linear embedding form, g:g(x_i)=W _gx_j, the overall process of a nonlocal block can be represented by Eq. (13). (13)${\displaystyle z_i=W_z{y}_i+x_i}$ where y_i is given in Eq. (11), W _z represents the weight matrix, and “+x _i” denotes the residual connection. The overall structure is illustrated in Fig. 5.

Artifact discriminant learning process

In image super-resolution, a LR input corresponds to many possible high-resolution (HR) outputs in a high-dimensional image space. The adversarial loss, Ladv, updates the results along multiple possible directions, leading to an unstable optimization process and resulting in artifacts and loss of details. This study introduces a method called artifact discrimination learning to preserve the true details and suppress artifacts while stabilizing the training of the generative adversarial network (GAN).

Firstly, we assume that the resolution of the input color image, I_SR, is H ×W ×3, and we use M(i,j) ∈[0,1] to represent the probability that I _SR(i,j) is an artifact pixel. The objective is to obtain a pixel-level mapping, M ∈RH ×W ×1. Since both artifacts and details belong to high-frequency image components, the residual between the ground truth image, I _HR, and the reconstructed image, I _SR, is first computed to extract the high-frequency components represented by Eq. (14). (14)${\displaystyle R=I_{HR}-I_{SR}.}$

As restoring fine texture information in the super-resolution process of images is more challenging, artifacts are often composed of overshoot pixel values. Therefore, the local variance of the residual map, R, is computed as the main indicator of artifact pixels. The calculation process is represented by Eq. (15). (15)${\displaystyle M\left(i,j\right)=\mathrm{var}\left(R\left(i-\frac{n-1}{2}:i+\frac{n-1}{2},j-\frac{n-1}{2}:j+\frac{n-1}{2}\right)\right)}$ where var represents the variance operator and n denotes the local window size and is set to 7.

Since the local variance is calculated with a very small receptive field, distinguishing artifacts from edges and textures can be unstable and may result in incorrect penalization of genuine details. To address this issue, a stable patch-level variance is computed based on the entire residual map, R, represented by Eq. (16). Typically, images with smoother regions have a smaller σ value when compared to images with rich texture details. By scaling the main mapping, M, by σ⋅ M, a more reliable artifact mapping can be obtained and denoted by (16)${\displaystyle \sigma \&={\left(\mathrm{var}\left(R\right)\right)}^{\frac{1}{q}}.}$

Furthermore, to further stabilize the training process and refine the artifact map, a dynamic gradient descent optimization is employed for the image super-resolution model, represented by Ψ. The exponential moving average (EMA) method is used to obtain a more stable model, Ψ_EMA, from Ψ, as shown in Eq. (17). (17)${\displaystyle {\Psi }_{EMA}^{\left(k\right)}\&=\alpha \cdot {\Psi }_{EMA}^{\left(k-1\right)}+\left(1-\alpha \right)\cdot {\Psi }^{\left(k\right)}}$ where α represents a weighting parameter set to 0.999. (⋅)^1/q scales the global variance, var(R), to an appropriate proportion, and in our experiments, q = 5 is set. The outputs of the two super-resolution models as I_SR1 = Ψ(I_LR) and I_SR2 = Ψ_EMA(I_LR) are denoted. Two residual maps, R₁ = I_HR − I_SR2 and R₂ = I_HR − I_DR2 are calculated and the artifact map, σ⋅ M, is defined by using Eq. (18). (18)${\displaystyle M_{refine}\left(i,j\right)=\left\{\begin{array}{c}0,if\left|R_1\left(i,j\right)\right|<\left|R_2\left(i,j\right)\right|\hfill \\ \sigma \cdot M\left(i,j\right),if\left|R_1\left(i,j\right)\right|\ge \left|R_2\left(i,j\right)\right|\hfill \end{array}\right.}$

The refined artifact mapping, M_refine, will only penalize locations where —R₁(i,j)— ≥—R₂(i,j)—. In locations where the residual of I _SR1 is smaller than that of I _SR2, the model, Ψ, updates in the correct direction and is not affected by the refinement process.

After the aforementioned calculations, the refined artifact map, M_refine, is obtained and the artifact discrimination loss, L_artif, is defined by Eq. (19). (19)${\displaystyle L_{artif}={∥M_{refine}\cdot \left(I_{HR}-I_{SR1}\right)∥}_1.}$ By introducing the artifact discrimination loss, L_artif, into the EGAN baseline model, the final loss function is represented by Eq. (20). (20)${\displaystyle L_G=L_{G_{-een}}+\beta L_{artif}}$ where β represents a weighting parameter set to 5 based on experimental results. The process of artifact discrimination learning is as follows: I_LR is fed into both the super-resolution models, Ψ and Ψ_EMA, resulting in outputs I_SR1 and I_SR2, respectively. Then, the original high-resolution image, I_HR, is used along with I_SR1 and I_SR2 to construct the refined artifact map, M_refine. The artifact discrimination loss, L_artif, is computed based on I_HR, I_SR1, and M_refine. Finally, the overall model loss, L_G, is used to optimize the model, Ψ, and the parameters of Ψ are aggregated over time to form Ψ_EMA. This process is repeated until the algorithm converges.