Optimizing colorectal cancer segmentation with MobileViT-UNet and multi-criteria decision analysis

Technical description

In this study, we have utilized publicly available EBHI-Seg dataset (https://doi.org/10.6084/m9.figshare.21540159.v1), comprises 4,456 histopathological images of colorectal cancer (Shi et al., 2023). This includes 2,228 histopathology section images and 2,228 corresponding ground truth images. The dataset covers five stages of tumor differentiation: Normal (76 images), Polyps (474 images), Low-grade intraepithelial neoplasia (IN) (637 images), High-grade intraepithelial neoplasia (IN) (186 images), Serrated Adenomas (58 images), Adenocarcinomas (795 images). Each histopathology section image is paired with a ground truth mask indicating the precise location of the abnormalities as determined by clinicians. The dataset, obtained through intestinal biopsy, featured images at 400× magnification using a Nissan Olympus microscope and NewUsbCamera software. The magnification details include a 10× eyepiece magnification and a 40× objective magnification.

Biological background

Normal colorectal tissue sections reveal regularly aligned tubular cells that appear unaffected characterized by structured morphology indicative of healthy tissue integrity (De Leon & Di Gregorio, 2001). In contrast, colorectal polyps, while similar in shape to normal structures, exhibit a distinct histological composition. Viewed under a microscope, polyps appear as redundant masses growing on the surfaces of body cells, often considered undesirable enlargements of the mucosal tissue (Cooper et al., 1998). Histopathological sections of polyps show intact luminal structures with minimal cellular division, primarily distinguished by a slightly higher atomic mass compared to normal tissues.

Intraepithelial neoplasia (IN) is a critical precancerous lesion characterized by increased branching of adenoid structures, dense arrangement, and varied luminal sizes compared to normal tissue. Cellular morphology shows enlarged nuclei with increased division (Ren et al., 2013). The Padova classification divides IN into low-grade and high-grade categories, with high-grade IN exhibiting more pronounced structural changes and greater nuclear enlargement than low-grade IN.

Adenocarcinoma is a malignant tumor of the digestive tract characterized by an irregular distribution of luminal structures. Identifying its boundary structures is challenging, and the nuclei are significantly enlarged at this stage (Jass & Sobin, 1989). Serrated adenomas are rare lesions, representing about 1% of all colonic polyps (Spring et al., 2006). The endoscopic appearance is not well defined but is believed to resemble that of colonic adenomas, particularly those with tubular or cerebral crypt openings (Li & Burgart, 2007). Figure 2 presents a differentiation tumor section along with the corresponding ground truth diagram.

VGG16-UNet

In this study, the VGG16-UNet model is employed for colorectal cancer segmentation, using VGG16 as the encoder to leverage its powerful feature extraction capabilities. VGG16, originally developed by Simonyan & Zisserman (2014), is well-regarded for its simplicity and effectiveness in deep learning tasks. It consists of 13 convolutional layers organized into five blocks, each followed by max-pooling layers that reduce spatial dimensions while retaining essential features. For use in segmentation, the fully connected layers of VGG16, designed for classification, are removed, allowing the architecture to integrate seamlessly as the encoder component in the VGG16-UNet model.

The decoder in VGG16-UNet closely resembles the vanilla UNet decoder, using up-sampling layers followed by convolutional layers to progressively refine the segmentation map. Skip connections are added from each encoder layer to its corresponding decoder layer, allowing high-resolution features from earlier layers to merge with the up-sampled features in the decoder, thus enhancing segmentation accuracy.

The final layer of the VGG16-UNet model consists of a single convolutional filter with a 1x1 kernel and a sigmoid activation function, producing the final segmentation output with pixel-level classification. This combination enables VGG16-UNet to benefit from VGG16’s deep feature extraction strength while maintaining the spatial detail necessary for precise segmentation through UNet’s architecture. Figure 3 illustrates the architecture of VGG16-UNet used in this study.

ResNet50-UNet

In this study, the ResNet50-UNet model is used for colorectal cancer segmentation, incorporating the ResNet50 architecture as the encoder to leverage its advanced residual learning capabilities. Introduced by He et al. (2016), ResNet50 is well-known for its use of residual connections, which help mitigate the vanishing gradient problem and support the construction of very deep networks. ResNet50 consists of 50 layers organized into multiple residual blocks, where shortcut connections bypass one or more convolutional layers. This structure facilitates improved gradient flow and feature learning across layers. In ResNet50-UNet, the output from ResNet50’s final layer before the classification block serves as the bottleneck layer, which is then passed to the decoder. The decoder component closely follows the standard UNet design, using up-sampling layers and convolutional layers to progressively restore spatial resolution. To further enhance segmentation accuracy, skip connections are established from each encoder layer to the corresponding decoder layer, merging high-resolution features with up-sampled feature maps. The final layer of the ResNet50-UNet model includes a single convolutional filter with a 1 × 1 kernel and a sigmoid activation function, producing the segmentation map with pixel-level classification.

This combined approach allows ResNet50-UNet to capitalize on the depth and residual learning strengths of ResNet50, achieving accurate segmentation by integrating high-level semantic information with spatial detail, as facilitated by UNet’s architecture. The architecture of the ResNet50-UNet model used in this study is shown in Fig. 4.

MobileNet-UNet

In this study, the MobileNet-UNet model is employed for colorectal cancer segmentation, using MobileNetV1 as the encoder to maximize efficiency through depthwise separable convolutions. MobileNetV1, introduced by Howard et al. (2017), is recognized for its lightweight architecture, which significantly reduces computational complexity and memory usage while maintaining strong performance. This makes MobileNetV1 especially suitable for deployment in resource-limited environments, such as mobile devices and edge computing platforms.

The encoder in MobileNet-UNet comprises 28 layers that utilize depthwise separable convolutions. This approach splits the convolution operation into two parts: a depthwise convolution, which applies a separate filter to each input channel, and a pointwise convolution, which combines these outputs. This method greatly reduces the number of parameters and computational cost compared to standard convolutions, while still effectively capturing important features from the input images.

The decoder in the MobileNet-UNet model follows the classic UNet structure, with up-sampling and convolutional layers to gradually reconstruct the segmentation map from encoded features. Skip connections link corresponding layers of the MobileNetV1 encoder to the decoder, merging low-level spatial details with high-level semantic information. This enhances segmentation accuracy by preserving fine-grained details.

The architecture of the MobileNet-UNet model used in this study is illustrated in Fig. 5.

MobileViT-UNet

In this study, we propose the MobileViT-UNet model, which incorporates MobileViT as the encoder within the UNet framework to improve segmentation of colorectal cancer histopathological images. MobileViT, introduced by Mehta & Rastegari (2021), combines vision transformers with convolutional layers to leverage the strengths of both approaches, enhancing performance in image segmentation tasks.

In the MobileViT-UNet model, the encoder uses both convolutional layers and transformer blocks. The convolutional layers capture local details and texture, essential for delineating the intricate structures found in histopathological images. Meanwhile, the transformer blocks capture long-range dependencies and global context, which are particularly useful for segmenting complex and varied tumor structures. This dual approach allows the model to integrate both detailed local features and broad contextual information. The decoder in MobileViT-UNet follows the standard UNet structure, using up-sampling and convolutional layers to reconstruct the segmentation map from the encoded features. Skip connections between corresponding layers in the encoder and decoder help retain spatial details and incorporate multi-scale features, further enhancing segmentation accuracy.

By integrating MobileViT within the UNet framework, this model combines the precise feature extraction of convolutional networks with the comprehensive contextual understanding provided by transformers. Figure 6 illustrates the MobileViT-UNet architecture.

Training of the models

The training process involves training each model twice using two different loss functions: Binary Cross-Entropy (BCE) and Dice. This methodology resulted in two distinct instances for each model. For instance, the VGG16-UNet model one instannce is trained with the BCE loss function, resulting in the instance referred to as VGG16-UNet+BCE. It was then trained with the Dice loss function, yielding the instance named VGG16-UNet+DICE. This procedure was consistently applied across all models. This approach is to compare the effectiveness of different loss functions within the same model architecture for colorectal histopathological image segmentation.

The Binary Cross-Entropy loss function is defined as: ${\displaystyle \text{BCE}=-\frac{1}{N}\sum _{i=1}^N\left(y_{i}log\left(p_i\right)+\left(1-y_i\right)log\left(1-p_i\right)\right)}$

where y_i represents the true label, p_i is the predicted probability, and N is the number of samples.

The Dice loss function is given by: ${\displaystyle \text{Dice Loss}=1-\frac{2\sum _{i=1}^N\left(y_i\cdot p_i\right)+ϵ}{\sum _{i=1}^N{y}_i+\sum _{i=1}^N{p}_i+ϵ}}$

where y_i represents the true label, p_i is the predicted probability, N is the number of samples, and ϵ is a small constant added to avoid division by zero.

Other training settings are: The Adam optimizer was used consistently across all configurations, with a learning rate of 0.0001 to maintain stable training dynamics and ensure fair comparisons. This learning rate was chosen empirically after testing various values, where 0.0001 was found to provide the best convergence for most models. Each model was trained with a batch size of 8, balancing computational efficiency with convergence speed. To reduce or to avoid over fitting and enhance model generalization, early stopping was implemented by monitoring the validation loss during training. Table 1 provides a comprehensive overview of the training configurations utilized for each segmentation models

Table 2 outlines the computational environment utilized for training the models. The training was conducted using Anaconda, with the TensorFlow and Keras frameworks. The hardware configuration included an NVIDIA RTX GPU with 6 GB of memory and 16 GB of RAM, providing robust computational power and ensuring efficient model training and performance.

Statistical analysis

The models’ effectiveness and robustness are evaluated using a non-parametric statistical test. MobileViT-UNet+Dice, identified as the top-performing model from Table 3, is compared pairwise against other methods using the Wilcoxon signed rank test (WSRT) with significance criteria p ≤ 0.05 as significant. The WSRT was applied on a per-image basis for each model pair across evaluation metrics—Dice ratio, Jaccard index, precision, and recall. This per-image approach captures variability across individual images, providing a more detailed assessment of model performance differences. The models per image performance across all metrics is provided as box plot in Figs. 7, 8, 9 and 10, along with annotation of pairwise WSRT significance.

The WSRT results for each metric are shown in Tables 4, 5, 6 and 7.

From Table 4, we observe that MobileViT-UNet+Dice performs significantly better on the Dice ratio than other models, with particularly comparable performance to MobileViT-UNet+BCE, where the results indicate similar efficacy. Table 5 highlights MobileViT-UNet+Dice’s strong results for the Jaccard index as well, showing significantly better performance than other models, further indicating its robustness in this metric.

In Table 6, MobileViT-UNet+Dice maintains high precision, significantly outperforming nearly all models except for MobileViT-UNet+BCE, with which it shares non-significant results, suggesting comparable precision performance. Likewise, in Table 7, MobileViT-UNet+Dice demonstrates strong recall scores, with significant differences in favor of this model over most alternatives. Notable exceptions are the comparisons with VGG16-UNet+Dice and MobileNet-UNet+Dice, where performance differences are non-significant.

Therefore the WSRT results across all metrics consistently reinforce MobileViT-UNet+Dice as a highly effective and robust model, sustaining superior or comparable performance relative to other models across each evaluation metric.

Multi-criteria decision analysis for ranking segmentation models

From the statistical analysis we observed some methods have no significant difference in performance across some evaluation metrics therefore it is difficult to select the best one. In order to identify the most best performing model, we conducted a multi-criteria decision analysis (MCDA). MCDA is a systematic approach for evaluating and ranking based on multiple criteria in complex decision-making scenarios. This method facilitates informed decision-making by balancing and considering multiple perspectives.

In this study, we employed an prominent MCDA technique i.e Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) combined with the entropy weighting methodology for our performance criteria. The assessment criteria that we used for assessment are Dice ratio, Jaccard index, precision, and recall. The result of TOPSIS and ranking of model is presented on Table 8. The TOPSIS scores acrross all tissue type are represent as bar graph in Fig. 11. Table 8 reveal that the MobileViT-UNet+Dice model achieved the highest TOPSIS scores across all classes therefore attaining the highest ranking among all models. MobileViT-UNet+BCE obtains the second ranking and ResNet50-UNet+BCE is ranked the lowest. This outcome underscores MobileViT-UNet+Dice superior performance in our study and supports its selection as the most effective model for segmenting colorectal regions in histopathology images.