Image reconstruction in graphic design based on Global residual Network optimized compressed sensing model

Overall design

Drawing inspiration from non-local mean value theory and leveraging deep learning technology, the author proposes an image CS reconstruction method based on non-local feature fusion network. The model is depicted in Fig. 2. To enhance the reconstruction of image block information, this article adopts an adaptive fusion of non-local a priori information using the image non-local mean theory, thereby enabling a cooperative representation of image information. The image structure information is constructed by designing a two-stage reconstruction network that progressively refines the image from coarse to fine.

During the enhancement reconstruction stage, the image non-local similarity prior plays a crucial role in providing complementary information to reconstruct individual image block features, thereby fully utilizing the internal structural features of the image. To this end, a non-local feature adaptive interaction module is designed, comprising two non-local feature fusion convolutions, a channel correlation discriminator module, and a spatial correlation discriminator module. This adaptive weighted fusion of non-local image features enables the suppression of mismatching information.

Collaborative reconfiguration group construction

The original image is first segmented into B × B blocks of non-overlapping size x_i, i = 1, 2, …, N and then the blocks are transformed into B² × 1 dimensional column vector, and use Gaussian random sampling matrix ϕ to obtain the compressed sampling values. y_i, whose observation matrix is Φ , then the signal measurement model can be expressed as Eq. (1). (1)${\displaystyle y_i={\Phi }_{x_i}.}$

When the observation matrix and the measurement results are known, the original graph matrix can be recovered using a one-dimensional parametric definition. (2)${\displaystyle x=min\parallel x{\parallel }_1}$ ${\displaystyle \mathrm{s}.\mathrm{t}.\mathrm{ }{y}_i=\Phi x_i.}$

Take B size 16, convert the image block to a B² dimensional column vectors, using the observation matrix ϕ for sampling.

The cosine similarity metric effectively captures the internal structural changes of an image while minimizing the impact of uneven illumination on the construction of similar blocks when measuring the similarity of two image blocks. Therefore, in this article, the cosine similarity of the compressed observations of the two blocks is computed to assess the similarity of their observations. By comparing the similarity of the compressed observations of a single image block with those of all other image blocks, the author ranks the obtained cosine similarity values from the largest to the smallest, selecting the compressed observations with greater similarity to form a cooperative reconstruction group Y _i.

The linear mapping network F was used to upgrade the dimension of the cooperative reconstruction group Y _i to obtain the initial reconstruction image block of the cooperative reconstruction group Z_i, as shown in Eq. (3). (3)${\displaystyle Z_i=F_1\left(W_1,Y_i\right),i\in \left[1,N\right]}$

where F₁ represents the fully connected layer. W₁ is the weight of the fully connected layer, which is obtained from the network training. F uses the fully connected network layer F₁ to upgrade the dimension and convert the compressed observed value in the collaborative reconstruction group Y _i, and obtains the initial reconstructed image block collaborative reconstruction group with the size of B × B.

The channel stitching technique is used to fuse the non-local similar features of the image. The initial reconstruction block co-reconstruction group Z_i contained in the initial reconstruction estimates of the image blocks z_i and its initial estimate of non-local similar features z_i,1, z_i,2, …, z_i,5. The joint channel stitching is performed to obtain the co-reconstructed features with fused non-local similar features Z_c, whose equation is shown in Eq. (4). (4)${\displaystyle Z_c=\text{concat}\left(z_i,z_{i,1},\dots ,z_{i,5}\right)}$

where concat means channel splicing.

Unlike traditional convolutional operations focusing on local neighborhoods, this module aggregates information from all positions, enriching the feature representations with a global context. It operates through non-local operations that compute responses at each position as a weighted sum of features from all other positions, with weights determined adaptively based on feature similarity. This process involves generating an attention map highlighting the significance of different feature positions, guiding the aggregation of features to focus on the most relevant parts of the input.

First, the reconstruction network uses a fully connected layer to increase the dimension of the input measurements and reconstruct them into smaller feature images. Then, a sub-pixel convolution module composed of a sub-pixel convolution layer, a 3 ×3 convolution layer, and a batch normalization layer is used to improve the image discrimination rate gradually, and the calculation process is shown in Eq. (5). (5)${\displaystyle I^{HR}=f^L\left(I^{LR}\right)=\mathrm{PS}\left(W_L\mathrm{\ast }{f}^{L-1}\left(I^{LR}\right)+b_L\right)}$

where I^LR is the low-resolution image, that is, the smaller feature map. I^HR is the reconstructed high-resolution image, which is the final reconstructed image. f^L is the output image of layer L, and the calculation process of PS function is shown in Eq. (6). (6)${\displaystyle \mathrm{PS}{\left(T\right)}_{x,y,c}=T_{\lfloor x/y\rfloor \lfloor y/r\rfloor ,c\cdot r\cdot \mathrm{mod}\left(y,r\right)+c\cdot \mathrm{mod}\left(x,y\right)}}$

where PS operation can transform the low-resolution image with size H × W × C⋅r² into the high-resolution image with size rH × rW × C by periodic transformation.

The reconstruction process of the reconstructed image $\tilde{x}$ (Eq. 7) shows where R is the depth reconstruction network. (7)${\displaystyle \tilde{x}=R\left(f^L\left(y\right)\right)=\mathfrak{P}\mathfrak{S}\left(f^{L-1}\left(y\right)\right).}$

Enhanced reconstruction based on global residual module

The concept of residual networks proves to be highly applicable in the image reconstruction phase, and this study introduces a novel approach by proposing a fully convolutional residual block. During the network training process, a crucial mechanism is employed wherein input image features are directly connected across layers to the output. This connectivity preserves the convolution-extracted image features and prevents their loss in the forward channel as the network architecture deepens. This approach accelerates the convergence of the network by maintaining the integrity of essential image features throughout the reconstruction process.

Simultaneously, the loss function is crucial in optimizing the image reconstruction effect. By comparing the residual information of the input image with each generated image during training, the loss function progressively minimizes the disparity between them. This iterative refinement process allows the network to continuously improve its ability to reconstruct images accurately, ultimately enhancing the overall quality of the reconstructed images.

By leveraging the power of residual networks, incorporating the novel full convolutional residual block, and effectively utilizing the loss function, this proposed approach effectively addresses the challenges associated with image reconstruction. Preserving image features, accelerated network convergence, and iterative refinement all contribute to achieving superior image reconstruction results. Since the image obtained in the initial reconstruction process is degraded compared to the original image, the image blocks are refined and reconstructed. The obtained cooperative joint reconstruction features are Z_c into a global residual reconstruction network consisting of non-local feature adaptive interaction modules F_lg global residual reconstruction network composed of a stack of non-local feature adaptive interaction modules F_r. The final output image is obtained by fusing the features ${\mathrm{z}}_{\mathrm{i}}^{{}^{\prime }}$ as shown in Eq. (8): (8)${\displaystyle z_i^{{}^{\prime }}=z_i+F_r\left(W_2,Z_c\right)}$

where z_i is the initial estimate of the image block, W₂ is the residual network parameter.

The concept of residual networks is leveraged in the image reconstruction stage, where a state-of-the-art full convolutional residual block is proposed. During network training, input image features are directly connected across layers to the output, thereby ensuring the preservation of features extracted through convolution. This approach efficiently circumvents the deepening of network structure that can result in the loss of image features in the forward channel, thus accelerating network convergence. Additionally, the loss function is utilized to compare the residual information of the input image with each training-generated image, continually decreasing the differences between them and ultimately enhancing the image reconstruction effect.

Figure 3 illustrates that the residual block comprises a feedforward network featuring three convolutional layers, accompanied by a constant jump link. where x denotes the input feature map, the residual function is denoted by F(x), and the output of the residual block is denoted by H(x). Therefore, the expression of the residual block is shown in Eq. (9). (9)${\displaystyle H\left(x\right)=F\left(x\right)+x.}$

When F(x) tends to zero, which is equivalent to the output of the residual block H(x) and the input x is extremely close to the input, and the difference is small, i.e., H(x) = x . This is called the constant mapping of the residual function; after training the network with many data sets, the optimal value of F(x). The optimal value of the residual block is obtained as the output of the residual block H(x). The residual block’s output is acquired after training the network on many data sets. During the backpropagation stage, the gradients utilized for training the three convolutional layers within the residual block are saved. Additionally, each convolutional layer is succeeded by a PReLU activation function, thereby augmenting the network model’s non-linear expression.

Loss function

To begin with, these images must be initialized by converting them from RGB space to YCrCb. Subsequently, the image space is transformed from RGB to image space, with the luminance channel being selectively chosen. The network is then trained with four distinct sampling rates: 1%, 4%, 10%, and 25%, correspondingly. Diverging from conventional approaches in the domain of compression-aware image reconstruction, the proposed network is jointly trained alongside a complete convolutional residual reconstruction sub-network, with the loss function defined by Eq. (10): (10)${\displaystyle L\left(W\right)=\frac{1}{T}\sum _i^T{∥f\left(x_i,W,K\right)-x_i∥}^2}$

where T denotes the number of channels of the feature map during network training, W denotes the material weight parameter in the pick-and-place network, K denotes the weight parameters in the full convolutional residual reconstruction sub-network, and the two parts are jointly trained in a unified way to train the whole network by back-propagating the Euclidean distance between the original image labels and the reconstructed images.

Experimental setup

The experiment employs the 91-images dataset (https://zenodo.org/records/11163729, 10.5281/zenodo.11163729) and the BSD300-train dataset (https://zenodo.org/records/10667264, 10.5281/zenodo.10667264) as the training dataset. The 1-dimensional column vector is normalized in the range (0,1) for each dimension to expedite network convergence. The input to the network is obtained by sampling image blocks cropped from each image using a Gaussian random matrix, with the luminance components of the image blocks being extracted as supervised labels during training. The observation matrix employed in this experiment comprises a Gaussian random matrix that satisfies finite isometric constraints, with the sampling rates set to {0.01, 0.04, 0.10, 0.25}. The model hyperparameters are presented in Table 1.

To assess the efficacy of diverse algorithms, grayscale images extracted from the BSD200-train dataset were utilized for testing, a Jupyter notebook with eight different solutions for common problems of digital image processing, including object recognition and binarization using an adaptative threshold. The evaluation metrics were peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM).

The experimental environment utilized in this study is detailed in Table 2.

Analysis of results

Figure 4 visually depicts the mean PSNR values obtained from the test images. Initially, at a learning rate of 0.001, the PSNR curve shows fluctuations as the epochs progress. This fluctuation can be attributed to the learning process as the model gradually adapts to the data. However, the curve exhibits a more consistent pattern as the learning rate decays. This signifies that the model is converging and reaching a stable state, where further training iterations have minimal impact on improving the PSNR values. The plateauing of the curve indicates that the trained model has achieved convergence across the four distinct sampling rates, demonstrating the effectiveness of the learning rate decay strategy in facilitating model stability and optimizing reconstruction quality.

To assess the effectiveness of the proposed network, a comprehensive comparison was made with five representative deep learning-based CS techniques, namely ReconNet, NL-MRN, ISTA-Net, SCSNet, and GMNet. The evaluation of these algorithms involved utilizing nine standard grayscale images, which served as the test bed for the analysis. The resulting performance metrics and findings are graphically represented in Figs. 5 and 6, providing a clear visual representation of the comparative outcomes achieved by each technique.

The proposed model demonstrates remarkable performance, surpassing other deep learning-based techniques, as substantiated by its consistently highest average PSNR and SSIM values across the four sampling rates. SSIM, being a more sophisticated image similarity evaluation index, provides deeper insights into the quality of the reconstructed images. Notably, the network architecture introduced in this study showcases superior SSIM results compared to the reference algorithms. When SR is 0.01, 0.04, 0.10, 0.25, the average PSNR is 4.87dB, 5.23dB, 6.12dB, 7.95Db higher than ReconNet. The average improvement over GMNet is 0.27dB, 0.74dB, 0.64dB and 0.76dB, showing superior reconstruction performance.

This achievement can be attributed to the novel non-local feature adaptive interaction module incorporated within the proposed model. By integrating non-local feature fusion convolution, this module effectively combines non-local information, allowing for comprehensive and accurate fusion of features. Moreover, it introduces weight adjustments to highlight critical high-frequency features while diminishing the influence of redundant low-frequency data. This is accomplished by utilizing the channel correlation discriminant and spatial correlation discriminant modules, enabling a more refined and discriminating feature representation.

Furthermore, the proposed model offers a unique advantage by doubly suppressing features that exhibit in-group matching errors. By identifying and suppressing these erroneous features, the model significantly enhances image reconstruction accuracy. This suppression mechanism adds layer of refinement to the reconstruction process, resulting in improved image quality and fidelity.

The proposed model outperforms existing deep learning-based techniques, as evidenced by its superior average PSNR and SSIM values across various sampling rates. The introduction of the non-local feature adaptive interaction module, coupled with the suppression of in-group matching errors, enables the model to achieve exceptional reconstruction accuracy. These advancements generate high-quality reconstructed images, solidifying the efficacy of the proposed network architecture for image reconstruction tasks.

In addition to evaluating the image reconstruction quality of different techniques, it is equally important to consider comparing their respective reconstruction times. Figure 7 provides insights into the average GPU runtime necessary for reconstructing nine images across various sampling rates, each with a size of 256  × 256.

It is worth noting that Fig. 7 exclusively focuses on the time consumed by the network during the image reconstruction process to ensure a fair and accurate comparison. By analyzing the GPU runtime, the author comprehensively understands the computational efficiency and speed of the different techniques, allowing the author to assess their suitability for real-time applications or scenarios with time constraints.

Analyzing Fig. 7, it becomes discernible that ReconNet exhibits a lower computational complexity than the other techniques. However, its reconstruction performance falls short regarding both PSNR and visual effects. On the other hand, NL-MRN showcases slightly improved reconstruction performance, albeit with a marginally longer reconstruction time compared to ReconNet.

Another noteworthy technique is GMNet, which achieves superior reconstruction quality while retaining the fastest reconstruction speed among the evaluated methods. Combining high-quality results and efficient processing makes GMNet a compelling option for image reconstruction tasks.

Furthermore, the proposed model is evaluated against SCSNet. Notably, at low sampling rates, the reconstruction time of SCSNet is nearly on par with that of the proposed model. However, as the sampling rate increases, the depth of the SCSNet network grows, resulting in a corresponding increase in reconstruction time.

Comparatively, ISTA-Net exhibits the longest image reconstruction time while delivering average reconstruction quality. In summary, compared to these deep learning-based image reconstruction methods, the proposed model outperforms the others in terms of image reconstruction time and overall reconstruction performance. Its superior balance of efficient processing and high-quality results make it a promising choice for various image reconstruction applications.