A super-resolution network using channel attention retention for pathology images

Overall structure

The overall architecture of the proposed model CARN is shown in Fig. 1. It was mainly constructed using a limited number of Channel Attention Retention Blocks (CARBs).

First, for each HR image I_HR, the corresponding LR image was generated I_LR using the bicubic algorithm. Most SISR works choose Bicubic as the main degeneration kernel (Lim et al., 2017; Zhang et al., 2018b; Zhang et al., 2018c), and Bicubic is closer to the true degeneration of histopathological digital images (Chen et al., 2020). Subsequently, these images were segmented into fixed-size patches as the inputs of CARN. Considering that pathology images from a whole slide image (WSI) have various image styles, such as converted RGB images differentiated from the color structure caused by the staining of physical sections and scanning machines, the network has begun by normalizing the images by channels. Specifically, for an image I_n with size of H × W in the dataset I containing N images, the normalization result $I_n^{norm}$ is obtained in Eq. (1): (1)${\displaystyle I_n^{norm}=\sum _{i=1}^H\sum _{j=1}^W\left[I_{n\left(i,j\right)}^R-I_N^{Rmean},I_{n\left(i,j\right)}^G-I_N^{Gmean},I_{n\left(i,j\right)}^B-I_N^{Bmean}\right]}$ where $\left[I_{n\left(i,j\right)}^R,I_{n\left(i,j\right)}^G,I_{n\left(i,j\right)}^B\right]$ are the R, G, and B value of image I_n at position (i, j), and $\left[I_N^{Rmean},I_N^{Gmean},I_N^{Bmean}\right]$ is the mean value of RGB channel of dataset I.

Subsequently, the data went through several CARBs, where a long-skip connection with a residual structure was included, inspired by Zhang et al. (2018b). It is found that simply stacking CARBs does not enable the network to perform better. The purpose of the long jump is to send the unconvolved low-level features in the image to the bottom of the network, supplementing the low-frequency information ignored by the multi-layer residual inside a CARB. At the end of the network, to generate I_HR with the required scale, the sub-pixel shuffle layer (Shi et al., 2016) is used to amplify the features. For the SR tasks of 2 ×, 4 ×, and 8 ×, we specify the magnification of the sub-pixel shuffle layer as 2 and build the upsampling module using 1, 2, or 3 groups of such layers, respectively. For the 3 × SR task, the number of features in multiple channels firstly increases to 9 times, and then the features pass through 1 group of sub-pixel shuffle layers with magnification of 3.

Channel Attention Retention Block (CARB)

Previous CNN-based SR methods treated all feature channels as equal, which is not in line with reality. Each filter in a convolutional layer has a local receptive field of the same size, and its output does not contain any contextual information, as shown in Fig. 2A. Thus, Zhang et al. (2018b) used the CA mechanism to resolve this contradiction, as shown in Fig. 2B. The existing CA methods usually assign different weights to each feature channel, while focusing on how to build a CA block using the spatial distribution of breast cancer pathology images. It is observed that high-frequency information in I_LR derive from the palisade or honeycomb arrangement of human organs for pathology images. Therefore, pathology images contain richer high-frequency information and more evenly distributed low-frequency information than real-world images. Pathology images in RGB format converted from a digital pathology images format suitable for medical imaging devices, have a narrower range of color distribution than real-world images due to the staining operation of cell sections.

As a result, in this study, a global average pooling was adopted to convert the channel-related global spatial information into channel descriptors, and use Sigmod as the end-gating function. Meanwhile, two convolutional layers were added to further strengthen the up-sampling and the down-sampling features after the features were pooled as 1 × 1. Subsequently, a ReLU (Nair & Hinton, 2010) layer was added between the two convolutional layers to further activate the gating result, as shown in Fig. 2C.

To preserve the beneficial features of a pathology image as much as possible in a CARB and reduce the loss caused by channel compression, further down-sampling channels were set in the proposed model. In detail, the number of channels C became $\sqrt{C}$ after the features were down-sampled. From the observation of most of the computer vision tasks using the CA mechanism, it is found that the value of Reduction significantly affects the training speed of deep networks. Although a minor Reduction value reduces the information loss caused by channel compression, it is not conducive to the convergence of a network and the stimulation of beneficial channels in the original feature map. On the other hand, increasing the Reduction value diminishes the computational load of a network to a certain extent. Hence, a wider down-sampling convolutional layer than the traditional CA mechanism was chosen to perform weight excitation and learn the weights in a wider feature distribution. This is due to the fact that SR tasks for pathology images require to average the weights over channels to offset the effects of the low inter-pixel variance and low-frequency regions that account for the vast majority of the areas in an image on features. In the following section, multiple ablation experiments demonstrate the rationality of the Reduction value.

Let the feature map F = (f₁, …, f_x, …, f_C) be the input features, that means the feature map H × W with depth C. The average pooling layer shrinks F to 1 × 1 to get the main representation of this channel f_g, as defined in Eq. (2): (2)${\displaystyle f_g=AvgPool\left(f_x\right)=\frac{1}{\mathrm{H}\times \mathrm{W}}\sum _{i=1}^H\sum _{j=1}^W{f}_x\left(i,j\right)}$ where f_x(i, j) represents the value of the (i, j) position in the x-th two-dimensional feature f_x, and AvgPool(.) denotes the global average pooling. In a CA Retention, feature up-sampling and feature down-sampling are respectively performed on the main representation of all channels F_g, to obtain the attention-retained feature F_CAR, as defined in Eq. (3): (3)${\displaystyle F_{CAR}=Sig\left(FeatUS\left(\delta \left(FeatDS\left(F_g\right)\right)\right)\right)}$ where δ(.) and Sig(.) represent the ReLU activation and Sigmod activation, respectively. FeatDS(.) and FeatUS(.) denote down-sampling convolution and up-sampling convolution for CA-retained features, respectively. The sampling multiples are set to $\sqrt{C}$ to get more learning caps for the CA Retention. This is different from existing SISR models for real-world images. Using the same settings as this study will not lead to a significant improvement in SR tasks for real-world images, but will introduce more information loss. At the end of the block, the residual component F_CAR will be adaptively re-amplified.

In each CARB, the residual connection is introduced from the beginning to the end of the block, so that each CARB inherits and fully utilizes the residual information brought by the previous basic module. Let the initial feature vector of the network be F₀, and the transfer process of residual information is defined in Eq. (4): (4)${\displaystyle F_m=R_m\left(F_{m-1}\right)=R_m\left(R_{m-1}\left(F_{m-2}\right)\right)=R_m\left(R_{m-1}\left(\dots R_1\left(F_0\right)\right)\right)}$ where F_m is the feature transformation of F₀ after m CARBs and R_m(.)the residual representation of the m-th CARB. Then for a CARN with n CARBs, the residual information is defined as Eq. (5): (5)${\displaystyle F_N=F_0+F_n.}$

This nested residual structure has a major positive effect on the tasks. The whole model (CARN) utilizes contextual information to improve objective metrics within a small network depth. Simultaneously, this allows the reduction of the number of CARBs in CARN as much as possible. N_CARB = 8 in the benchmark model is set, to prove the efficiency and effectiveness of the setting in Section 4.

Loss function

Multiple CNN-based SR models chose MSE as the optimization loss of their networks because MSE usually leads to higher scores in the evaluation metrics such as PSNR. The MSE Loss is defined in Eq. (6): (6)${\displaystyle L_{MSE}=\frac{\sum _{i=1}^{rH}\sum _{j=1}^{rW}{\left\{I_{\left(i,j\right)}^{HR}-{\theta }_G{\left(I^{LR}\right)}_{\left(i,j\right)}\right\}}^2}{r^2\times H\times W}}$ where θ_G(.) represents the network optimization process; r, H, and W denote the number of samples, the number of pixels of sample height, and width, respectively. It is mentioned in Ledig et al. (2017) that once the magnification exceeds 4, I_LR outputted by an MSE-optimized network lacks high-frequency details and appears to be extremely smooth in terms of texture while maintaining high-objective metrics. This is not in line with the subjective thinking direction of human beings when judging whether the image is clear or not. In terms of objective metrics, PSNR or SSIM are not highly consistent with the perceptual quality and human subjective visual effects. Lim et al. (2017); Zhang et al. (2018b); Zhang et al. (2018c) used the L1 loss, as defined in Eq. (7), to improve the models’ performance, as Blau et al. (2018) pointed out that the L1 loss could tolerate larger errors compared to the MSE loss. Moreover, in terms of texture, the L1 loss enables networks to reproduce more high-frequency details than the MSE loss. (7)${\displaystyle L_{L1}=\frac{\sum _{i=1}^{rH}\sum _{j=1}^{rW}{∥{\theta }_G{\left(I^{LR}\right)}_{\left(i,j\right)}-I_{\left(i,j\right)}^{HR}∥}_1}{r^2\times H\times W}.}$

In most scenarios such as medical diagnosis, humans do not calculate the distance between two images pixel by pixel, but rather judge the structural similarity between them. A measure based on structural similarity was proposed in Wang et al. (2004), as defined in Eq. (8): (8)${\displaystyle L_{SSIM}=1-SSIM\left[{\theta }_G\left(I^{LR}\right),I_{\left(i,j\right)}^{HR}\right]}$ where SSIM(., .) represents the structural similarity measure of two samples. A loss function L_CARN for CARN is designed, as defined in Eq. (9), which is a linear combination of L1 loss, MSE loss, and SSIM loss. (9)${\displaystyle L_{CARN}=\alpha L_{L1}+\beta L_{MSE}+\gamma L_{SSIM}}$ where α, β, andγ are the weights of L1 loss, MSE loss, and SSIM loss, respectively.

In the conducted experiments, it is clear that when (α, β, γ) = (0.8, 0.1, 0.1), CARN showed the best performance by grid search. Therefore, L_L1 is the main component of the total loss function, enabling CARN to show better convergence and ensuring better subjective visual effects of pathology image SR results. L_MSE improves the pixel-level accuracy and reduces fluctuations in objective metric values. During the training process, the change of L_SSIM is extremely small (0.0974−0.0979 in CARN’s 2 × SR task), which not only enhances the structural similarity of the output results but also fixedly enlarges the difference between the testing samples and training samples to improve the model’s optimization level.

Dataset: bcSR

The dataset CAMELYON (Litjens et al., 2018) originated from two pathology challenges organized by the Diagnostic Image Analysis Group (DIAG) and the Department of Pathology of the Radboud University Medical Center in Nijmegen, the Netherlands in 2016 and 2017. The goal of these challenges was to evaluate new and existing algorithms for automatic detection and classification of breast cancer metastases in WSI images of breast cancer sentinel lymph node sections.

WSI is a technique for multi-scale digitization of traditional slides at various resolutions and is commonly used in the field of pathology cell images. High-performance WSI scanners are usually very expensive. It is a highly feasible, efficient, and relatively inexpensive strategy to use SR technology to enlarge LR images from WSI, replacing the process of WSI scanning HR images. Thus, data were collected from CAMELYON17 and a benchmark dataset bcSR was established to facilitate SISR applications on pathology images of breast cancer.

Deep CNN networks cannot directly use the WSI file format as experimental samples and require efficient sampling of information-rich and representative patches. Li & Ping (2018) sampled 400 initial WSI files within the CAMELYON dataset and provided 220,000 pre-sampled coordinate pairs and five levels of sampling resolution for cancer and non-cancer regions, respectively. 1,000 pre-sampled coordinates were selected from each of the two regions and they were sampled from the original level (level 0) of the WSI file with a block size of 1, 024 × 1, 024.

Considering that the cell tissue cannot cover the entire slice plane tightly when using certain coordinate positions for sampling to make a digital image, multiple areas without cell tissue will be acquired, such as sampling at the edge of the entire slice. Image areas without cellular organization have little effect in the SR task, and these areas appear white in the digital images. Consequently, to improve the content of useful information in the dataset, the procedure is as follows: To begin, convert all digital images to grayscale images; Afterwards, binarize the image with the grayscale threshold of 224; Finally, calculate the proportion of white pixels (with a grayscale range of 225–255) in each image, and delete images with more than 60% of the white pixel area.

On the basis of the previous step, the RGB mean value R_i of all pixels in each image and the RGB mean value R_all of all pixels in all images were separately calculated. All the images were ranked according to the distance from R_i to R_all. Finally, the 1,200 images with the largest distance were chosen as the experimental dataset, called bcSR. This operation further amplified the pixel differences between samples.

Figure 3 describes the construction process of bcSR. The newly constructed dataset is beneficial for improving the objective metrics of deep CNN networks, making networks observe more regions, and training diverse samples to enhance the SR capability of the networks. Meanwhile, compared with random sampling, the proposed sampling strategy significantly improves the efficiency of SR tasks.

Training details

The bcSR was split using 1000 images as the training set and 200 images as the test set for all models. The mean values of R, G, and B channels in training set are [0.7204, 0.4298, 0.6397]. LR images were generated by bicubic down-sampling. SRCNN (Dong, Loy & Tang, 2016), SRGAN (Ledig et al., 2017), EDSR (Lim et al., 2017), RDN (Zhang et al., 2018c), RCAN (Zhang et al., 2018b), and SWD-Net (Chen et al., 2020) were selected as comparison models. The network depth of EDSR, RDN, and RCAN were appropriately adjusted to match the total amount of CARN’s parameters (see Table 1).

For CARN, N_CARB = 8 was set; the filter size of all convolutional layers was 3 × 3, and the number of channels was equal to 64, except that the number of channels in the middle layer of the CA Retention was 8 (Reduction = 8). During training, all samples were randomly rotated by 90 degrees to achieve the purpose of data augmentation and split the training samples into 64 × 64 patches as input and 48 × 48 in the 3 × up-sampling task. The ADAM was used to optimize the network. The learning rate was initialized to 10⁻⁴ and halved after every 2 × 10⁵ minibatch update.

We use PSNR and SSIM to evaluate the performance of all SR models. PSNR is usually used to measure the distance between the final processed result and the original image, and its unit is ’dB’. The larger the PSNR, the lower the distortion of result. SSIM is usually used to estimate the similarity between two digital images. The range of SSIM is [0, 1]. The closer the SSIM is to 1, the more similar the two images are. For the experiments of all models in quantitative evaluation, the random seeds were respectively set to 1, 7, 11, 18 and 1011 to conduct the 5 groups of experiments. The average value of all experiments was taken as the final result of objective metrics.

The proposed network was implemented on the torch 1.8.0 platform and used a NVIDIA GeForce RTX 3090 (24GB) and a NVIDIA TITAN Xp (12GB) for all experiments. SRCNN and SRGAN are not able to run on the used device due to the problem of the original code platform, and SWD-Net only provides a 2 × up-sampling model in the original code. Thus, this study re-wrote and re-implemented these models.

Quantitative experiments

This study presents objective metric results on 2 ×, 3 ×, 4 ×, and 8 × up-sampling tasks for all models in Table 2. Compared with other methods, the CARN achieved the state-of-the-art PSNR/SSIM in all tasks and had an objective metric gain of 0.188/0.0203 over the second-place SWD-Net. It is worth noting that using bcSR to train deep CNN models such as RDN, RCAN, SWD-Net, and CARN, the objective metric results in the 2 × up-sampling task surpass all current published SR works based on real-world images or pathology images, strongly demonstrating the adaptability of the bcSR in the SR tasks. RCAN and RDN are verified on Set5, Set14, B100, Urban100 and Manga109 datasets, and their best results come from Manga109. In 2 × task, RCAN achieves 39.61/0.9788 and RDN achieves 39.38/0.9874 in PSNR/SSIM. The best result of EDSR comes from Set5: 38.20/0.9606. The performance of these competitive SR models in 3 ×, 4 × tasks is lower than that of real-world image SR tasks: the best results of PSNR are higher than 34dB (3 ×) and 31dB (4 ×). Clearly, the SR task of histopathology images is more challenging than that of real-world images. In the 8 × task, the performance of all models drops significantly. SWD-Net is specially designed for the SR task of histopathology images. The performance of the model on the 2 × task in the HistoSR dataset (also from CAMELYON) is 32.769/0.9510, and the corresponding performance of EDSR is 32.676/0.9502. CARN can maintain a stable performance output in high-level upsampling tasks, and maintain an advantage in the reconstruction of texture details.

To demonstrate the effect of the loss function L_CARN, the suggested network was retrained with MSE loss (see CARN(MSE) in Table 2). The performance of the proposed model with MSE loss on 2 ×, 3 ×, or 4 × SR task was slightly lower than the model itself. And the higher the magnification, the smaller the contribution of MSE loss is for SR image reconstruction. Furthermore, the CARN network was tested by adding a portion of samples—that have been removed during the bcSR construction, since they contain more than 60% blank area and have a short pixel channel distance—into the test set (see CARN* in Table 2). According to the results, the proposed network still shows similar or even better performance in these low-frequency regions, even though the removed images do not participate in the training process of CARN. This is exactly what people want to see. These results further demonstrate the satisfactory quality of this newly created dataset bcSR.

Figure 4 shows the average reconstruction time taken by all methods to process an image. The difference in efficiency mainly originates from the network depth and the optimization method. The suggested method is 29% faster than the second place in the 2 × task, and the average time consumption is 15% faster than the second place in all up-sampling tasks.

Qualitative experiments

Figure 5 shows the qualitative results of the CARN on 2 × and 4 × up-sampling tasks, respectively. The image details obtained by the proposed model far exceed other models, especially in some high frequency regions.

Ablation experiments

Influence of loss function

Figure 6 shows the effect of the loss function on the network’s performance (PSNR) for all up-sampling tasks. It is clear that the convergence speed and final performance of CARN using the suggested combined loss are superior to those of CARN with commonly used MSE loss in all tasks. Since the area and spread of high-frequency information in pathology images are larger than those in general images, for SR tasks, the aforementioned loss performs better in terms of the model’s convergence speed and objective metric results.

For CARN, exclusively using MSE loss will cause severe oscillation in the early training period, which is greatly obvious in the 2 ×, 3 ×, and 4 × up-sampling tasks. Using the above-mentioned loss function enables CARN to have high stability throughout the training phase and maintain the same or even a faster convergence rate than the MSE loss.

Moreover, this study trained the CARN with the SSIM loss, only. The results show that the SSIM loss fails to converge the CARN: the PSNR on 2 ×, 3 ×, 4 ×, and 8 × tasks are 12.341 dB (at epoch 119), 12.099 dB (at epoch 121), 12.124 dB (at epoch 19), and 4.072 dB (at epoch 1), which is quite different from the baseline results.

Table 3:

Investigations on the number of CARBs.

NCARB	2	4	8	12	16	20
PSNR(dB)/SSIM	39.780/0.9526	40.137/0.9792	40.385/0.9839	40.446/0.9846	40.456/0.9845	40.513/0.9846
FLOPs	3.981 G	6.421 G	11.300 G	16.179 G	21.058 G	25.936 G
Avg. Time (s)	18.7	26.5	41.5	59.0	76.2	90.6

DOI: 10.7717/peerjcs.1196/table-3

Results on binary classification for image diagnosis task

To further evaluate the role of CARN in the diagnosis task, this study chose ResNet-50 (He et al., 2016) as the baseline classifier and conducted the histopathology image classification task on the 2 × SR results of the bcSR and PCam (Veeling et al., 2018) datasets. ResNet is one of the classic structures in the field of computer vision and has been widely used for tasks such as object classification. The PCam dataset is derived from Camelyon16, which contains 327,680 patches extracted from slides of 10 × magnification (at 0.972 µm per pixel). Their size is 96 × 96. In the classification task, considering the input size of the ResNet-50 network, each image was segmented in bcSR into 64 images of 128 × 128 size. These segmented images maintained the same class labels as the original ones. The classification results are shown in Table 5. The deep learning-based SR models both achieved higher gains in performance than bicubic. Notably, CARN achieved the best results in both the bcSR and PCam datasets, with accuracies 11.56% and 20.80% higher than bicubic, and 1.62% and 2.78% higher than the second-ranked SWD-Net. The results for the image classification task show that the aforementioned CARN is able to recover more details from pathology images, thus facilitating downstream tasks such as real-time image diagnosis.