Edge and texture aware image denoising using median noise residue U-net with hand-crafted features

Convolution neural network for image processing

Due to a strong capacity to learn, CNNs have been formed for image denoising. The original distribution of noise in corrupted images may vary as a result of convolution operation, which could make training for image denoising more challenging. The batch normalization technique can address the problem by normalizing the input layers of a network. This article focuses on the internal covariant shift problem but the vanishing or exploding gradient may be severe (Ioffe, 2015). For retrieving smooth edges, the constrained total variation (TV) approach requires Fast gradient-based algorithms that can handle edges and eliminate noise in a corrupted image but may cause optimization problems (Beck & Teboulle, 2009).

To address additional inverse problems, denoiser prior can be promoted into model-based optimization techniques as a modular component (Zhang et al., 2017b). Alternatively, the introduction of BRDNet architecture, which integrates two networks to expand the width, addresses the mini-batch and internal covariant shift problems, reduces the training challenges of the structure, and improves performance. However, the main disadvantage of the model is more complex (Tian, Xu & Zuo, 2020). A dual denoising network (DudeNet) was introduced to restore clean images. A dual network with a sparse mechanism can bring out additional characteristics to improve the relevancy of the denoiser. The combination of local and global characteristics can take out salient features to restore clear details to complicated noisy images (Tian et al., 2020b). A framework for training DDM on a single image named SinDDM and learning the internal data of the training image utilizes a multi-scale diffusion procedure. They employ a fully convolutional lightweight denoiser, conditioned at both noise level and scale, to encourage backscatter. With this architecture, patterns of any size, from coarse to fine, can be created (Kulikov et al., 2023). The use of the medical field has seen many advancements. A wide range of image-denoising methods designed with medical imaging in mind are covered in this thorough review. The effectiveness of a wide range of contemporary and conventional techniques, such as wavelet transforms, non-local means, and deep learning approaches, is assessed in various medical imaging modalities, such as magnetic resonance imaging (MRI), computed tomography (CT), and ultrasound (Kaur & Dong, 2023). Dong, Ma & Basu (2021) present a novel feature-guided CNN that is intended for denoising ultrasound images that are captured from portable devices. Because of their portable nature and their noisy operation, ultrasound machines frequently yield images of inferior quality. The feature-guided CNN improves image quality while maintaining clinically significant details by utilizing domain-specific features like edges and textures (Dong, Ma & Basu, 2021). Dong & Basu (2023) describe a denoising method for medical images that uses an explainable AI (XAI) framework to maintain significant features while denoising. To guarantee that the final images maintain crucial diagnostic features while lowering noise, the authors suggest a novel loss function that combines feature preservation with conventional denoising goals.

Deep CNN for image denoising

A lot of research has been published using different methods of CNN, Tian et al. (2022) proposed that multi stage image denoising CNN with wavelet transform (MWDCNN) is a technique that consists of three stages: dynamic convolutional block (DCB), wavelet transform and enhancement block (WEB)s, and Residual block (RB) to improve the performance of denoiser. To overcome the width and depth limitations of lightweight CNNs and achieve better denoising performance, a dynamic convolution is used in the CNN and does not focus on complex validation (Tian et al., 2022). A new method of self-supervised image noise canceling that integrates a multi-masking approach with BSN (MM-BSN). It can be employed to resolve the issues of high noise cancellation, which other BSN methods cannot solve effectively (Zhang et al., 2023b). Res-WCAE with the Kullback-Leibler divergence (KLD) rule has been proposed for a lightweight and powerful deep learning architecture, particularly designed for denoising the fingerprint image. Res-WCAE contains two encoders and a decoder. It requires a large database to identify the correct fingerprint (Liang & Liang, 2023). Further, a data enhancement technique called recorrupted-to-recorrupted (R2R) has been proposed, to solve the problem of overfitting due to a lack of realism (Pang et al., 2021). Neshatavar et al. (2022) proposed a cyclic multi-variate function-supervised image denoising framework. The CVF-SID method can remove clear and noisy image maps from the input by exploiting several self-supervised loss metrics. For better denoising performance, deep boosting denoising net (DBDnet) has been developed. A noisy observation, creates a noise map from a residual learning network. To be more precise, it creates a raw noise map using a straightforward design, and then gradually upgrades the noise map using an enhancement function but the noise prediction may not be clear (Ma et al., 2022; Quan et al., 2021). The authors look into the possibility of complex validating CNNs for image denoise. To fully utilize the benefits of complex-valued operations, CNN was designed with its main operations defined in the complex number domain. The fact that the various models employed in this comparison study were not rigorously trained in identical settings is one of its limitations.

Moreover, the above-discussed models are increasing the performance by upgrading the networks at different levels. But it leads to complexity and more time consumption. CNNs are the most useful apparatus for denoising an image, as demonstrated by the methods-based CNNs discussed above. As a consequence, we follow suit and use a CNN to solve the image-denoising problem by constructing a simple architecture.

Basic U-Net architecture

The U-Net models are widely used for segmentation and denoising applications. The traditional model of U-Net architecture is shown in Fig. 1. The basic architecture of the U-Net model contains an encoder and decoder constructed only using the fully convolutional layers without using any dense layers. The encoders are equipped with convolution and downsampling done by kernel filters and pooling layers respectively. In contrast to encoders, decoders are constructed using up samplers and concatenation of the features from the encoder stage. Unlike the classification models, the final output is a denoised image instead of a class label.

The major strength of deep learning models relies on their ability to learn the feature automatically without human intervention. Though the conventional U-Net models could perform better for low-complex databases, the performance of U-Net is poor in the case of challenging applications such as denoising. Therefore, it was proposed to add hand-crafted features to the U-Net model. The U-Net model was constructed using the traditional encoders in the feature abstraction stage and the decoders were employed in the feature reconstruction stage.

MNRU-Net architecture

The MNRU-Net model was constructed using the traditional encoders in the feature abstraction stage and the decoders were employed in the feature reconstruction stage. The traditional model of U-Net architecture as shown in Fig. 1 was modified to the MNRU-Net model as depicted in Fig. 2. The main novelty in this work was the inclusion of hand-crafted features in the deep learning model.

The idea of this research work is to supply manual features to enhance the learning capability of the conventional U-Net model. Though improved versions of U-Net models are available, the main drawback of those models is the high computation cost. In addition, the noise pattern learned by the traditional U-net model is insufficient to reconstruct a better-denoised image.

Hence, the noise pattern obtained from the conventional median filter was applied in addition to the noisy gray image. Additionally, the gradient information of the image is also applied to the model. Hence, two image priors are evaluated and concatenated to the input gray image.

Let $I\left(x,y\right)$ be the clean image and $N\left(0,\sigma \right)$ be the noise. Then the noisy image I is represented as shown in Eq. (1).

(1) $$I_n\left(x,y\right)=I\left(x,y\right)+N\left(0,\sigma \right).$$

The denoised image $I_d$ from the median filter is obtained as Eq. (2). The value of the denoised image $I_d$ at pixel location $\left(x,y\right)$ is indicated by $I_d\left(x,y\right)$. It is the result of applying the median filter at this particular pixel. The noisy image $I_n$ at pixel location $\left(x-k,y-l\right)$ is represented by the expression $I_n\left(x-k,y-l\right)$. The median is computed over a local neighborhood centered around the pixel $\left(x,y\right),$ as indicated by the coordinates $\left(x-k,y-l\right).$

(2) $$I_d\left(x,y\right)=Median\left(I_n\left(x-k,y-l\right)\right)$$where the median represents the operation that computes the median value, and $\left(x-k,y-l\right)$ iterates over the pixels in the filter window around the pixel $\left(x,y\right).$ A 3 × 3 size window is used to filter the Gaussian noise. An approximate version of the noise matrix is obtained as shown in Eq. (3).

(3) $$Noise\left(x,y\right)=I_n\left(x,y\right)-I_d\left(x,y\right).$$

Let $I_d\left(x,y\right)$ be the recovered image from the median filter and gradient magnitude image $G\left(x,y\right).$ The gradient magnitude is obtained as given in Eq. (4).

(4) $$G\left(x,y\right)=\sqrt{\left(G_x{\left(\mathrm{x},\mathrm{y}\right)}^2+G_y{\left(\mathrm{x},\mathrm{y}\right)}^2\right)}$$where vertical gradient G_y $\left(x,y\right)$ and Horizontal gradient G_x $\left(x,y\right)$ are calculated using Sobel operators as follows:

(4a) $$\begin{array}{rl}{G}_x(x,y)=& (I_d(x+1,y-1)+2I_d(x+1,y)+I_d(x+1,y+1))\\ & -(I_d(x-1,y-1)+2I_d(x-1,y)+I_d(x-1,y+1)\end{array}$$

(4b) $$\begin{array}{rl}{G}_y(x,y)=& (I_d(x-1,y-1)+2I_d(x,y-1)+I{I}_d(x+1,y-1))\\ & -(I_d(x-1,y+1)+2I_d(x,y+1)+I_d(x+1,y+1))\end{array}$$

By substituting Eqs. (4a) and (4b) in Eq. (4) we get a gradient image. Hence, the MNRU-Net model is applied with three layers of information namely, the gray image, the predicted noise image from Eq. (3), and the gradient image from Eq. (4).

The structure of the encoder used in this research is shown in Fig. 3. All the convolutional layers were activated by the activation function of the rectified linear unit (ReLU) activation function. However, the final convolution layers were activated by using the ‘sigmoid’ function w hich yielded the final denoised image. Drop-out layers were also added in convolutional layers of both decoder and encoder to improve the performance of denoising. A dropout of 0.05 was done in the decoder and encoder stages. The encoder and decoder sections are constructed using the convolution blocks C1, C2, C3, and C4 as shown in Figs. 3 and 4, respectively. Each of the convolution blocks is internally consisting of two convolutional layers. The number of filters in each encoder block is 64, 128, 256, and 512 respectively for blocks 1, 2, 3, and 4, respectively, in downstream.

Traditional deep networks may experience gradients that explode or vanish as well as becoming stuck in weak local minima when the learning rate is too high. These problems are aided by batch normalization. Normalizing network activations prevents slight differences in layer parameters from amplifying as the data spreads throughout a deep network (Ioffe, 2015).

Handcrafted features

When applying hand-crafted features to image-denoising tasks, neural networks perform much better, especially when there is a lack of training data or a high noise level. By using pre-defined features that capture significant image properties, the model can make use of handcrafted features, which minimizes the need for large datasets and intensive augmentation. For situations where data availability is restricted, MNRU-Net becomes especially helpful. In contrast to more intricate and sophisticated models, MNRU-Net achieves competitive outcomes with a more straightforward architecture. Handcrafted features reduce the need for deeper layers, making the model faster to train and less expensive to compute.

As demonstrated by the improved performance metrics (PSNR, SSIM, figure of merit) across various noise standard deviations, U-Net’s architecture combined with hand-crafted features allows for better handling of high noise levels.

Loss function

Since the problem is related to the reconstruction of the noiseless image, the loss function was chosen as mean squared error (MSE). The loss function as shown in Eq. (5) was used during the training session.

(5) $$MSEloss={\displaystyle \frac{1}{M\times N}\sum _{i=1}^M\sum _{j=1}^N{\left(predicte{d}_{i,j}-actualtarge{t}_{i,j}\right)}^2}$$where N = No of Columns, M = No of Rows, and i, j are pixel locations, predicted_,j = denoised images during training session, actual target_i,j = the Original Noise free image. The main metrics were to reduce MSE loss and obtain the least error.

Metrics evaluation

The learning rate of the model was initially set at 1 × 10⁻³. This rate was adaptively modified based on the validation performance. The learning rate was reduced in non-linear steps of 1 × 10⁻⁵, 1 × 10⁻⁶ and the least of 1 × 10⁻⁷. L2 regularization was set to 0.000001. This adaptive variation was done using the call-back functions when there was no improvement even after a patience of 10. The model was optimized by utilizing the Adam optimizer (Kingma & Ba, 2014). The batch size was adjusted to four in the case of the BSD300 database. This model was trained for 50 epochs. In the training loop, the sessions were iterated through four images per batch to cover all the training images. Since the main aim is to improve the learning capability of the traditional U-Net model, image augmentation was not done in this work.

The training metric was chosen as the peak signal to noise ratio (PSNR) as shown in Eq. (6).

(6) $$PSNR=10\log {\displaystyle \frac{Ma{x}^2}{MSEloss}\mathrm{d}\mathrm{B}}$$where Max = Maximum value of the pixel in the original noise-free image.

The testing performance was obtained from the images that were not exposed to training and validation. The average performance in terms of MSE and PSNR was obtained as 15,25 and 50 dB respectively. The performance of the MNRU-Net model to denoise is evaluated based on PSNR, SSIM, and FOM. In addition to the traditional metrics, a few of the feature-based metrics were also included in the performance evaluation. The formula to calculate the metrics is presented in Eqs. (7) & (8).

(7) $$FOM={\displaystyle \frac{1}{max\left(\left|G_t\right|\left|D_c\right|\right)}\sum {\displaystyle \frac{1}{1+k.d_{G_t}^2\left(p\right)}}}$$ (8) $$SSIM\left(x,y\right)={\displaystyle \frac{\left(2{\mu }_x{\mu }_y+c_1\right)\left(2{\sigma }_{xy}+c_2\right)}{\left({\mu }_x^2+{\mu }_y^2+c_1\right)\left({\sigma }_x^2+{\sigma }_y^2+c_2\right)}.}$$

Training and testing datasets

The images were obtained from the Berkeley Segmentation Dataset (BSD). The images were of dimensions 481 × 321 and 321 × 481, with three color channels R, G, and B. The gray format of images was used after resizing those to 400 × 400, with an option of nearest interpolation. The database of 300 images was split into 80% of training images which accounted for 240 images. The remaining images of 30 and 30 were used for validation and testing respectively. The split of the database was done after shuffling the names of the images with a random state of 42. All the images were initially rescaled to in the range [0, 1] from its original scale of [0, 255].

For testing purposes, we used a set of four benchmark datasets they are Set12, KODAK24, McMaster, and CBSD68. The description of each dataset is given below.

Set12: A compact collection of 12 grayscale pictures that are frequently used to assess denoising algorithms, enabling a direct comparison with techniques that are currently in use in the literature.

Kodak24: This collection of 24 finely detailed color images is perfect for evaluating how well denoising techniques maintain color fidelity and sharpness.

McMaster: 18 color images with an emphasis on textures and high-frequency details, which is essential for determining how well fine-grained image details are preserved when denoising.

CBSD68 is a subset of the Berkeley Segmentation Dataset made up of 68 color images that were chosen especially for testing denoising algorithms on a variety of real-world images with different content.

Advantages and disadvantages of MNRU-Net

By using handcrafted features in addition to learned features, MNRU-Net improves denoising performance, particularly in high-noise scenarios. This results in the denoised image’s edges and textures being better preserved. MNRU-Net maintains a simpler architecture when contrasted with more intricate deep-learning models. Its simplicity makes it more accessible and faster to train, especially on constrained hardware, by lowering computational costs and memory requirements. In conclusion, MNRU-Net has a lot to offer in terms of improved performance and decreased complexity even with small amounts of data, but it also has drawbacks in terms of feature design, generalization, and possible redundancy with learned features.