SEU²-Net: multi-scale U²-Net with SE attention mechanism for liver occupying lesion CT image segmentation

Multi-scale feature extraction

Multi-scale feature extraction has emerged as a pivotal technique in the field of computer vision and image analysis. This approach recognizes the inherent complexity and diversity of visual information present in images, and addresses it by extracting features at varying scales. The utilization of multi-scale feature extraction enables the model to capture intricate details, from fine-grained textures to larger contextual structures, thus facilitating robust and comprehensive understanding of the input data.

Over the years, a multitude of methods have been proposed to integrate multi-scale information, ranging from pyramidal structures to more advanced deep learning architectures that dynamically learn features at different resolutions. For pyramid in multi-scale feature extraction, Lin et al. (2017) introduced the concept of Feature Pyramid Networks (FPN), revolutionizing multi-scale feature extraction in object detection. FPN establishes lateral connections between feature maps from different layers, combining high-resolution details from lower layers with high-level semantics from upper layers. Subsequently, Zhao & Zhang (2021) discussed the Enhanced Multi-scale Feature Fusion Pyramid Network for object detection, which emphasizes more effective fusion of multi-scale features, leading to heightened precision in object detection. Dong, Zhang & Qu (2021) presented the Multi-path and Multi-scale Feature Pyramid Network (MM-FPN) for object detection. This network likely explores multi-path feature extraction while integrating multi-scale representation within the pyramid structure. Lastly, Jiao & Qin (2023) proposed the Adaptively Weighted Balanced Feature Pyramid for object detection. This approach potentially introduces adaptive weighting within the pyramid structure to balance feature contributions and enhance object detection.

U-Net and its variants have achieved tremendous success in biomedical image segmentation, closely tied to their multi-scale feature extraction architecture. The U-Net’s encoder–decoder structure allows feature extraction at different scales. The contracting path captures high-level semantics, while the expansive path restores spatial details. This combination excels in delineating objects of varying sizes, making U-Net suitable for tasks like liver lesion segmentation. The Nested U-Net architecture extends U-Net by nesting multiple U-Net blocks within each other. This nested design further enhances multi-scale feature extraction by enabling the network to capture features at various levels of abstraction, improving the model’s capability to distinguish objects of diverse sizes and complexities. Gudhe et al. (2021) introduces a novel multi-level dilated residual neural network. By incorporating multi-level dilated residual blocks and non-linear residual blocks into skip connections, the network achieves enhanced multi-scale feature extraction capability, leading to improved performance in biomedical image segmentation. Dense U-Net  (Cheng et al., 2021; Aldoj et al., 2020; Cao et al., 2020) introduces dense connections in the decoder, linking each layer to all preceding layers, enhancing feature propagation. This architecture aids in capturing multi-scale features and improving the identification of edges and details.

Attention mechanism

In recent years, attention mechanisms have gained significant prominence across various deep learning tasks, including image segmentation. These mechanisms enhance the network’s ability to focus on relevant regions within feature maps, thereby improving both accuracy and interpretability. We introduce self-attention mechanism with both spatial attention and channel attention. The self-attention mechanism can establish relationships between all positions in the input sequence, regardless of a fixed window size and employ multiple heads to simultaneously capture features from different attention perspectives. The Transformer (Vaswani et al., 2017) and its variants (Zhang et al., 2019; Mor & Winquist, 2002; Liu et al., 2021; Fu et al., 2019) utilize the self-attention mechanism, whicn can be modeled for use in sequence-to-sequence tasks. It employs a multi-head self-attention mechanism to capture relationships between different positions within input sequences, thereby enabling modeling of long sequences. The spatial attention mechanism focuses on selectively weighting different spatial positions within an image. By calculating the similarity of each position, it adaptively assigns weights to different spatial locations, enabling the model to emphasize target boundaries and details. In contrast, the channel attention mechanism directs its focus towards different channels within a feature map. By recalibrating the relationships between channels, the channel attention mechanism adaptively emphasizes channels that are more important for the task, while reducing reliance on irrelevant channels. For example, Bottleneck Attention Module (BAM) and Convolutional Block Attention Module (CBAM) incorporate both spatial and channel attention mechanisms. The Squeeze-and-Excitation (SE) block introduces a channel attention mechanism to adaptively recalibrate feature maps.

Liver occupying lesion CT image segmentation

Numerous techniques have been utilized for liver occupying lesion segmentation  (Anter & Abualigah, 2023), with a specific emphasis on liver tumor segmentation. As outlined in the Introduction section, the segmentation of liver-occupying lesions can be classified into three main approaches, which rely on grayscale segmentation algorithms, machine learning, and deep learning. Table 1 summarizes the machine learning and deep learning methods applied to liver occupying lesion segmentation.

Residual U-block with squeeze-and-excitation

Residual U-block with Squeeze-and-Excitation (SE-RSU) is a new structure introduced in U²-Net as shown in Fig. 1, which incorporates the SE mechanism into the RSU of U²-Net. The SE-RSU terms that the SE block is added to the position of the residual connection in the RSU. The SE block is an attention mechanism used to enhance the model’s feature representation capability by adaptively learning the correlations between feature channels to better capture important feature information, as shown in Fig. 2. The purpose of adding SE after the first convolutional block of SE-RSU is coarse-grained context detection in the initial layer network and fine-grained context detection in the deep network as SE-RSU goes deeper in the SEU²-Net network.

Figure 2 shows that the SE block is implemented by passing the output of the convolution operation through a global pooling layer, two fully-connected layers (FC) and and activation function operations in SE-RSU. Firstly, global spatial information compression into channel descriptors is achieved by using global average pooling to generate channel statistics. i.e., the size H × W × C feature is compressed to 1 × 1 × C. The role of the fully connected layers is to restrict model complexity, promote generalization, and perform linear transformations to facilitate the activation function in capturing inter-channel correlations more effectively. Then, the ReLU and sigmoid activation function is used to learn nonlinear interactions between channels to capture channel dependencies. The output of sigmod function is channel-multiplied with the original feature to highlight the useful feature information and ignore the less useful ones. Finally, residual connection in the SE block is used to avoid gradient vanishing. Equation (1) shows that W₁ and W₂ represent the weights of two FC layers, and ReLU(x) is defined as max(0, x).σ represents the sigmoid activation function. The H, W and C stand for the height, width, and depth (number of channels) of the feature map respectively, and X_c(i, j) denotes the value of the feature map at position (i, j) of the c_th channel. (1)${\displaystyle SE=\sigma \left(W_2\cdot max\left(0,W_1\cdot \frac{1}{H\times W}\sum _{i=1}^H\sum _{j=1}^W{X}_c\left(i,j\right)\right)\right)\odot X_c\left(i,j\right)+X_c\left(i,j\right).}$

Loss functions

The standard binary cross-entropy loss function is used to describe U²-Net, however, we use a combination of CrossEntropy and Dice Loss functions in our approach. The Dice Loss function is primarily used to optimize the model’s prediction accuracy and robustness, particularly for enhancing the prediction precision and robustness of the model. The CrossEntropy Loss function is renowned for its ability to measure pixel-level class probability discrepancies. By using these two loss functions together, the model can find a balance between accuracy and precision and has better generalization ability. However, the binary cross entropy loss function is sensitive to pixel class distribution imbalance, which can cause the model’s prediction results to be biased towards regions with more pixel classes, and can result in misclassification in regions with fewer pixel classes. Additionally, the binary cross entropy loss function cannot handle the problem of target boundaries well, which can lead to prediction results with breaks or blur at the boundaries, affecting the segmentation performance of the model. Therefore, the binary cross entropy loss function is not effective for small or complex occupying lesion contours in liver occupying lesion segmentation.

Choose to use a mix of CrossEntropy and Dice loss functions with corresponding weights of 0.7 and 0.3. Equation (2) is the loss function expression of CrossEntropy Loss. N represents the number of samples, C represents the number of classes, y_ij represents the jth label of ith sample, and p_ij represents the predicted probability of the jth class for ith sample. The cross-entropy loss function measures how close the predicted value is to the ground truth. (2)${\displaystyle CrossEntropyLoss=-\frac{1}{N}\sum _{i=1}^N\sum _{j=1}^C{y}_{ij}\ast log\left(p_{ij}\right).}$

Equation (3) is the expression of the loss function for Dice Loss, where Dice is given by Eq. (7). The Dice Loss function is used when the background area is much larger than the target area. Because the liver occupying lesion area is smaller than the liver background area, the Dice loss function will have a better effect. (3)${\displaystyle DiceLoss=1-\frac{|A\cap B|}{\left|A\right|+\left|B\right|}.}$

Evaluation metrics

IoU refers to the ratio between the intersection and union of the true value and the predicted value in a class. The formula is as shown in Eq. (4). Here, TP represents the intersection of the predicted liver occupying lesion and labeled occupying lesion. TP + FP + FN represents the union of the predicted liver occupying lesion and labeled occupying lesion. (4)${\displaystyle IoU=\frac{TP}{TP+FP+FN}.}$

Accuracy (Acc) is the overall classification accuracy, which is the probability of predicting the correct number of samples over the total number of samples. Acc is expressed in Eq. (5) using confusion matrices. The intersect_area represents the intersection area of prediction and ground truth on all classes. All classes in the experiment refers to the liver space-occupying lesion and background. pred_area represents the prediction area on all classes. (5)${\displaystyle Acc=\frac{intersect\text{\_}area}{pred\text{\_}area}.}$

The Kappa coefficient measures whether the predicted value is consistent with the actual classification value. The Kappa coefficient compensates for the bias towards large categories in Acc caused by the gap in the number of categories. Equation (6) is the calculation method of Kappa coefficient, where P_e represents the ratio of the product of the number of true categories and the number of corresponding predicted categories to the square of the total number of samples. A Kappa coefficient between 0.81 and 1 indicates almost perfect agreement. (6)${\displaystyle Kappa=\frac{Acc-P_e}{1-P_e}.}$

The Dice coefficient represents the proportion of duplicate parts between the predicted segmented image and the annotated image. Dice has a value between 0 and 1. In Eq. (7), A represents the pixel value of the true image and B represents the pixel value of the predicted image. (7)${\displaystyle Dice=\frac{|A\cap B|}{\left|A\right|+\left|B\right|}.}$

Process of experiment

The datasets were divided into the training set, test set, and verification set in a ratio of 8:1:1, respectively. The Cross Entropy and Dice Loss functions are combined. The momentum optimizer which contains the Newtonian momentum flag can better eliminate the wobble phenomenon in the process of updating the hyperparameter. We use cosine annealing methods to dynamically adjust the learning rate as shown in Eq. (8). (8)${\displaystyle {\eta }_t={\eta }_{min}^i+\frac{1}{2}\left({\eta }_{max}^i-{\eta }_{min}^i\right)\left(1+cos\frac{T_{cur}\pi }{T_i}\right)}$

Let i denote the number of learning rate changes at the i_th time. The initial learning rate ${\eta }_{max}^i$ is 0.0015. The minimum learning rate ${\eta }_{min}^i$ is 0. T_cur is the current epoch. T_i is the number of iterations in one learning rate cycle, which is set to 5400. The batch size is 4. The model is saved every 900 training runs, for a total of 9,000 training runs. The experiments are trained on an NVIDIA Tesla V100 GPU (32 GB).

PUFH dataset

Dataset setup

The PUFH dataset consisted of 200 3D abdominal CT images and corresponding labeled images. The images ared labeled to highlight various types of liver occupying lesions such as hepatic cysts, liver abscess, and hepatocellular carcinoma. The number of slices that can be scanned at a time ranges from 39 to 107 depending on the information of each 3D image when the dataset is sliced horizontally. The resolution of the slice images ranged from (0.5839844, 0.5839844, 5.0) to (0.88671875, 0.88671875, 5.0) mm, and the image intensity ranged from (−1024.0, 3071.0) to (−3024.0, 1210.0). The slice size is 512 × 512. Figure 3 shows the fusion slice image of a random abdominal CT image and its corresponding annotated image in the PUFH dataset using ITK-Snap.

Experimental results

We compared SEU²-Net, an improved model derived from U²-Net and a member of the U-Net family, with U-Net, U²-Net, and its variants, including AttentionU-Net (Oktay et al., 2018), UNet3Plus (Huang et al., 2020), and UNetPlusPlus (Zhou et al., 2020). Additionally, to assess SEU²-Net’s superior performance, we compared it with other recent state-of-the-art semantic segmentation techniques, including DDRNet _23 (Pan et al., 2022), DNLNet (Yin et al., 2020), and GSCNN (Takikawa et al., 2019). Table 2 shows the evaluation metrics of PUFH dataset in nine models. SEU²-Net outperforms for liver CT image segmentation in five methods, achieving the IoU ratio increased 0.44%, the Acc increased 0.02%, the Kappa coefficient increased 0.26%, the Dice coefficient increased by 0.24% over U²-Net. Compared with AttentionU-Net, which is the combination of attention mechanism and UNet, SEU²-Net had the highest improvement, among which the IoU ratio increased by 8.49%, the Acc increased by 0.34%, the Kappa coefficient increased by 5.08% and the Dice coefficient increased by 4.9%. When compared with DDRNet _23, DNLNet, and GSCNN, it is evident that SEU²-Net consistently outperforms these models by a considerable margin.

Table 2:

Evaluation metrics table of PUFH dataset.

	IoU (%)	Acc (%)	Kappa (%)	Dice (%)
U-Net	86.85	99.54	92.72	92.96
AttentionU-Net	82.06	99.36	89.81	90.14
UNet3Plus	65.10	98.53	78.10	78.86
UNetPlusPlus	81.92	99.42	89.76	90.06
U2-Net	90.11	99.68	94.63	94.80
SEU2-Net (Ours)	90.55	99.70	94.89	95.04
DDRNet _23	83.90	99.49	90.98	91.24
DNLNet	83.99	99.48	91.03	91.30
GSCNN	84.22	99.49	91.17	91.43

DOI: 10.7717/peerjcs.1751/table-2

Notes:

Bold styling indicates the maximum or highest values.

Figure 4 shows that six 2D slices are randomly selected in the dataset to observe the liver occupying lesion segmentation effect of the nine models. The effect of liver occupying lesion segmentation depends on the location of segmentation, the overall contour and the recognition of small liver occupying lesion. SEU²-Net is most similar to label in terms of the location and size of liver occupying lesion. For example, the liver occupying lesions in the fifth slice has complex contours and fine regions. Comparing the segmentation effects of the six models in the fifth slice, SEU²-Net can identify very small liver occupying lesion and complex edges (the mark of the red box), which greatly illustrates the importance of SEU²-Net’s attention mechanism and its ability to capture feature information from multiple scales perspectives.

LiTS dataset

Dataset setup and processing

Liver tumors are a type of liver lesion, therefore, the LiTS dataset of 131 contrast-enhanced 3D abdominal CT scans includes 70 scans for training and testing with corresponding annotated images specifically focusing on liver occupying lesion segmentation. The number of tumors varied from 0 to 75, and the size varied from 38 mm³ to 349 mm³. The layer spacing (section thickness) is (0.45,6) mm. The number of slices that can be scanned at a time ranges from 42 to 1,026 according to the different information of each 3D image. The size of 2D slices is 512 × 512, where the planar resolution is (0.6 × 0.6, 1.0 × 1.0) mm.

For the processing of the LiTS dataset, only the tumor label is retained during liver tumor segmentation because the dataset contained both liver and tumor labels. Then, the processing of the dataset is the same as that of the PUFH dataset. Figure 5 shows the fusion slice image of a random abdominal CT image and its corresponding annotated image in the LiTS dataset.

Experimental results

The nine models trained on the PUFH dataset are used for training and testing in the LiTS dataset. The evaluation metrics of LiTS dataset are shown in Table 3. Compared with U²-Net, SEU²-Net has the IoU increased by 2.7%, Kappa coefficient increased by 1.65%, and Dice coefficient increased by 1.63% in the evaluation metrics. UNet3Plus performs the worst among the evaluation metrics. Compared with UNet3Plus, the IoU of SEU²-Net is increased by 20.11%, Acc is increased by 0.23%, the Kappa coefficient is increased by 13.55%, and the Dice coefficient is increased by 13.44%. At the same time, the evaluation metrics of SEU²-Net on LITS dataset are still higher than those of AttentionU-Net. When comparing SEU²-Net against non-UNet variants within the LiTS dataset, its outstanding performance is clearly demonstrated. Figure 6 shows six randomly selected 2D slices in the LiTS dataset to observe the liver tumor segmentation effect of the nine models. Most of liver tumors in these six pictures are small, and SEU²-Net is most similar to label in terms of the location and size of liver tumors, which proves that SEU²-Net has advantages in the segmentation of small tumors.

Table 3:

Evaluation metrics table of LiTS dataset.

	IoU (%)	Acc (%)	Kappa (%)	Dice (%)
U-Net	75.57	99.78	85.98	86.08
AttentionU-Net	75.98	99.78	86.24	86.35
UNet3Plus	63.26	99.63	77.31	77.49
UNetPlusPlus	73.46	99.75	84.57	84.70
U2-Net	80.67	99.83	89.21	89.30
SEU2-Net (Ours)	83.37	99.86	90.86	90.93
DDRNet _23	74.87	99.78	85.52	85.63
DNLNet	64.72	99.68	78.42	78.58
GSCNN	69.37	99.72	81.78	81.91

DOI: 10.7717/peerjcs.1751/table-3

Notes:

Bold styling indicates the maximum or highest values.

Model complexity

SEU²-Net exhibits outstanding performance on both datasets in terms of evaluation metrics. However, in order to evaluate the model complexity, we conduct a comprehensive analysis of this model. This crucial for gaining a more extensive understanding of the model’s characteristics, as it encompasses two key components: computational cost, typically quantified in floating point operations (FLOPs), and the number of model parameters (Params). In this section, we delve into the model complexity of SEU²-Net and compare it with several state-of-the-art semantic segmentation models, including UNet, AttentionU-Net, UNet3Plus, UNetPlusPlus, U²-Net, DDRNet _23, DNLNet, and GSCNN.

Table 4 presents a comparison between SEU²-Net and eight other models in terms of FLOPs and Params for network complexity.The maximum values in FLOPs and Params are highlighted in bold in the table. It becomes evident that SEU2Net demonstrates a comparatively higher level of computational complexity. This is primarily attributed to the inherently greater complexity of the U2Net model itself. When comparing the model complexities of SEU²-Net and U²-Net, it is evident that SEU²-Net’s computational complexity is only marginally higher than that of U²-Net. Specifically, SEU²-Net exhibits an increase of 1,152 FLOPs and 1,184 Params compared to U²-Net. This marginal increment in complexity can be attributed to the incorporation of SE blocks into all eight SE-RSU blocks within SEU²-Net, resulting in an additional computational load of 148 FLOPs and 144 Params for each SE block. Despite this slight increase in computational demand, SEU²-Net achieves a noteworthy 1% enhancement in IoU on the LiTS dataset compared to U²-Net. This emphasizes the effectiveness of SE blocks in boosting segmentation performance while maintaining the model’s relative lightweight characteristics.

Loss function analysis

In the context of liver occupying lesion segmentation, the choice of an appropriate loss function plays a crucial role in determining model performance. We employed a combination of CrossEntropy and Dice Loss functions, denoted as mixture loss. Mixture loss offers a synergistic advantage by promoting prediction precision and robustness in segmentation tasks. To ascertain the superiority of our novel hybrid loss function over each individual loss function, we conducted the loss function ablation experiment. Additionally, we consider binary cross-entropy loss function (BCE Loss) suitable for binary segmentation tasks, since it is adopted by the authors of U²-Net as the loss function. To provide a comprehensive evaluation, we compared against CrossEntropy Loss, Dice Loss, and BCE Loss individually in the PUFH and LiTS dataset. Detailed experimental results can be found in Tables 5 and 6. The highest value in the evaluation metrics is highlighted in bold.

It becomes evident that mixture loss outperforms all other loss functions, including the CrossEntropy, Dice, and BCE Loss functions, across all evaluation metrics (IoU, Acc, Kappa, Dice) for both the PUFH and LiTS datasets. This consistent superiority in segmentation performance on both datasets demonstrates that Mixture Loss excels in accurately capturing the characteristics and boundaries of liver occupying lesions. To visualize the evolution of loss functions during model training, Figs. 7 and 8 illustrate the changing curves of the four loss functions throughout the training process, accompanied by the corresponding fluctuations in mIoU on the validation dataset. Based on the two images, we can observe that when Dice Loss is used independently, there is a phenomenon of loss function non-convergence and a very poor performance in terms of evaluation metrics. The use of CrossEntropy or BCE Loss functions individually leads to fewer oscillations in the curves, it is evident that Mixture Loss outperforms all others in terms of all evaluation metrics.

Limitations and future work

Although SEU²-Net performs well, there are still some limitations in this study. One key limitation lies in the relatively large number of parameters associated with SEU²-Net when compared to other state-of-the-art methods. This parameter abundance can pose challenges in resource-constrained environments, such as on mobile devices or embedded systems. Future research should focus on optimizing the SEU²-Net architecture to reduce its parameter count while preserving its essential multiscale and attention mechanisms. This optimization aims to make SEU²-Net more adaptable for various medical image segmentation tasks, especially in resource-limited settings. Furthermore, this study’s dataset primarily concentrates on liver occupying lesions. Expanding the evaluation of SEU²-Net to encompass larger and more diverse datasets is essential to ensure its generalizability across a broader spectrum of medical image segmentation tasks. Extending the evaluation to diverse datasets will enhance our understanding of SEU²-Net’s capabilities and limitations in various medical imaging scenarios.

Future studies should focus on optimizing the SEU²-Net architecture to reduce parameter count while preserving its key features is essential. We plan to study the effects of different SE block configurations and compare them with other attention mechanisms such as CBAM (Woo et al., 2018) and BAM (Park et al., 2018). Conducting extensive evaluations on diverse datasets will further validate SEU²-Net’s applicability in various medical image segmentation scenarios, enhancing its practical utility in the field.