Enhancing colorectal polyp classification using gaze-based attention networks

Classification of colorectal polyps based on CNNs

The classification of early colorectal polyps mainly relies on manual identification by endoscopists. This method is very time-consuming and cumbersome for physicians, and it also creates difficulties in ensuring the accuracy of classification due to the subjectivity of manual identification. The need for endoscopists to carefully observe and judge a large quantity of images can easily cause fatigue and affect the accuracy of diagnosis. In traditional research methods, researchers often use feature extraction technology to collect information such as shape, texture, and color from polyp images. These features can provide information for identifying and classifying different types of polyps. However, the feature extraction process is complex and requires researchers to have considerable professional knowledge and experience. Additionally, the extracted features must be further processed and screened to ensure their effectiveness for performing the classification task (Patel et al., 2020). With the improved performance of deep neural networks, CNNs are increasingly used in the classification of colorectal polyps. Compared with traditional manual feature extraction methods, CNNs can more efficiently extract abstract and high-level features, thereby improving the accuracy of classification. A CNN-based system can automatically process and analyze a high volume of endoscopic images, significantly reducing the doctor’s review time. In addition, CNN-based methods reduce the risk of misdiagnosis and underdiagnosis, providing patients with more timely and accurate treatment options (Sánchez-Peralta et al., 2020).

Various advanced CNNs have emerged to aid in the classification of colorectal polyps. Liu et al. (2023) proposed Polyp DeNet, which combines dilated convolution and efficient channel attention modules based on ResNet-50 and uses parameter sharing and data augmentation methods to improve model performance. This model has achieved a high level of accuracy in colorectal polyp classification, fully demonstrating the benefit of artificial intelligence-assisted diagnosis. Sharma et al. (2023) proposed a unique lightweight CNN with a discrete wavelet pooling strategy for colon polyp classification. This model uses wavelet pooling instead of ordinary pooling layers to achieve a better balance between receptive field size and computational efficiency. Rahman, Wadud & Hasan (2021) used an integrated model combining Xception, ResNet-101, and VGG-19 networks to achieve higher accuracy in classifying hyperplastic, serrated, and adenomatous polyps, as well as non-polyps. Tanwar et al. (2022) proposed a single-shot multiframe detector (SSD) architecture with image preprocessing and added dropout in the fully connected layers to prevent overfitting and improve the generalization ability of the model. This assisted in effectively detecting and classifying colorectal polyps in colonoscopy images. Hossain et al. (2023) proposed an autonomous CRC screening method to detect polyps and assess their potential threat. In that study, DoubleU-Net was used for polyp segmentation, and Vision Transformer (ViT) was used for classification according to polyp risk. The proposed method classified polyps in the Endotech Challenge and Kvarsi-SEG datasets as either hyperplasic or adenomatous with a test accuracy of 99%.

Although deep learning models play an important role in colorectal polyp classification, and CNN-based diagnostic models have shown excellent performance, they still have some limitations. CNNs are data-driven algorithms that require a large amount of labeled data. The annotation process of datasets is very cumbersome, labor-intensive, and time-consuming, posing a challenge to doctors. Additionally, these models have poor interpretability, and doctors require more information to guide their decision-making process. In a medical setting, these models must maintain high accuracy and provide clear explanations of the predicted results.

Eye-tracking-based medical image analysis

Eye-tracking technology provides a potential solution to the problems of cumbersome annotation and insufficient interpretability of CNNs. By tracking the eye movements of doctors while they view endoscopic images, this technology can provide valuable data for developing more interpretable and transparent classification models. These data help reveal key features and areas in colorectal polyp images, thereby improving the interpretability of the model. Therefore, combining eye-tracking technology with CNNs may be a promising way to improve colorectal polyp classification.

In recent years, eye-tracking technology has been widely applied in the medical field as auxiliary prior knowledge to assist in training more efficient CNN models. Stember et al. (2019) proposed a dynamic annotation method based on eye-tracking for lesions and organs in multimodal medical images including computed tomography (CT) and magnetic resonance imaging (MRI) and applied eye-tracking information to automatic image annotation. Karargyris et al. (2021) proposed a multi-head model based on U-Net that incorporates radiologists’ gaze maps for chest X-rays. The clinical-expert gaze maps generated by eye-tracking equipment can be used as an alternative to weakly supervised labels to guide the training of deep learning models. This method was used to simultaneously predict the expert’s attention patterns and their classification results to distinguish between congestive heart failure, pneumonia, and healthy tissue. Wang et al. (2022) proposed a gaze-guided attention model based on X-ray images to classify osteoarthritis into four categories with good interpretability and classification performance. Another study (Wang et al., 2024) proposed a gaze-guided graph neural network (GNN). As a real-time, realistic, end-to-end disease classification algorithm, GNN integrates raw eye gaze data and can achieve high-accuracy classification without preprocessing. Ma et al. (2023) proposed a novel and effective saliency-guided visual transformer (SGT) model that utilizes eye movement data to correct ViT shortcut learning. This model can effectively learn and use human prior knowledge, perform well in multiple datasets, and improve the interpretability of the ViT model. Wang, Zhang & Ge (2023) proposed a method that combines the gaze information of radiologists with image labeling requirements. A dual-path encoder is used to integrate gaze information, and a cross-attention transformer module is used to embed the gaze patterns of doctors reading images into the network model. Through multi-feature skip connections, spatial information is combined in the downsampling process to offset the internal details of the segmentation, achieving the high-precision segmentation requirements. Jiang et al. (2024) proposed an eye-tracking-based early diabetic retinopathy (DR) detection model using ophthalmologists’ gaze maps. The weighted gaze map is integrated as a supervision mask to guide the learning of the DNN model’s attention. A novel difficulty-aware and class-adaptive gaze-map attention learning strategy was proposed to enhance the interpretability of the model. The attention-guided approach regularized by the Class Activation Map (CAM; Zhou et al., 2016) shows improvements in the accuracy and interpretability of early DR detection models. The key information of these studies is summarized in Table 1.

These studies highlight the potential of eye-tracking technology in the field of medical imaging diagnosis. Eye-tracking technology not only acts as auxiliary prior knowledge to aid in training deep learning models but also enhances the interpretability and comprehensibility of these models. By integrating eye-tracking data, researchers gain deeper insights into the attention and decision-making processes of medical experts, thereby refining the training strategies and performance of deep learning models. Moreover, eye-tracking technology can serve as an alternative weakly supervised label, guiding the training of deep learning models and thereby enhancing their accuracy and credibility in medical imaging diagnosis.

Eye-tracking data collection process

Standardized data collection procedures were used for the collection of high-quality eye-tracking data. The main body of the eye-tracking device used in this study included a 1,920 × 1,080 27-inch high-definition liquid crystal display (LCD) and a Tobii Eye Tracker 5 eye-tracking device. The eye-tracking device connected to an external computer via a data cable and was used to record binocular gaze data.

This study implemented a customized data collection software in Python (Wang et al., 2022) that quickly constructed the experimental environment through simple parameter settings. Eye-tracking data was collected from endoscopists at a frequency of 90 Hz. Endoscopists sat on a fixed chair and kept their eyes perpendicular to the screen before providing eye movement data that simulated clinical working conditions. Following the user manual of the Tobii Eye Tracker 5 eye-tracking device, the distance between the endoscopists’ eyes and the screen was adjusted within the range of 50 to 80 cm. The eye-tracking device calibration was personalized for each doctor to account for different reading habits, which helped adapt to the vision of different individuals and reduce eye-tracking errors. Each clinician performed a nine-point calibration procedure during first use of the eye-tracking device.

During the diagnosis of colorectal polyps, each image was displayed in the center of the LCD screen, and the endoscopist entered their diagnosis using the numeric keys on the keyboard. For example, pressing the number “1” indicated that the polyp type was diagnosed as Type 1, and “Enter” indicated that the diagnosis was completed, prompting the next image. During this process, the eye-tracking device continuously recorded the participant’s eye movement information including the saccade point, fixation point, fixation duration, and eye scanning path, all of which was stored in the computer.

Gaze attention heatmap generation

During the eye-tracking data collection process, a series of gaze points on each diagnostic image was recorded in real-time. However, despite personalized calibration, slight deviations in the endoscopists’ gaze points could still occur due to the inherent system error of the equipment. To reduce these deviations, a Gaussian function G (x, y) (shown as Eq. (1)) was applied to transform each gaze point into the gaze area.

(1) $$G\left(x,y\right)={\displaystyle \frac{1}{\sqrt{2\pi }\sigma }{e}^{\frac{-\left[(x-x_c{)}^2+(y-y_c{)}^2\right]}{2{\sigma }^2}}.}$$

In this equation, (x_c, y_c) represented the central gaze point, and the pixel level distance of variance σ was used as the effective field of view range. In this study, a higher σ represented a broader range of focus for the endoscopists’ attention, potentially encompassing areas unrelated to the diagnosis. However, a lower σ could impede the ability of the clinician to concentrate on a specific lesion area, resulting in attention being divided among several locations inside a single lesion and raising computational requirements. It is appropriate to use a value of σ that modestly compensates for human visual error. The following proportion equation was developed:

(2) $${\displaystyle \frac{\pi {({\sigma }_1)}^2}{H\cdot W}={\displaystyle \frac{\pi {\sigma }^2}{H_p\cdot W_p}.}}$$

H_P = 1,080, W_p = 1,920 represented the display resolution, and H = 48 and W = 64 were the screen physical values corresponding to H_P and W_p, respectively. According to geometric operations, σ₁ = πRθ/360°, in which θ represents perspective error. The pixel variance σ on the display was estimated as:

(3) $$\sigma ={\displaystyle \frac{\theta }{{360}^{\circ }}\pi R\sqrt{{\displaystyle \frac{H_p\cdot W_p}{H\cdot W}}}.}$$

In this equation, R is the approximate distance area of 50 to 80 cm between the eyeball and the fixation point on the screen. Inspired by Jiang et al. (2024), θ was set to 1° to offset most of the errors. As specified in the user manual for the Tobii Eye Tracker 5, the maximum angular deviation for a gaze point was approximately $\pm {0.5}^{\circ }$. Corresponding to the interval $[-{0.5}^{\circ },{0.5}^{\circ }]$ Finally, the determined σ value fell between 17.1 (R = 50 cm) and 27.2 (R = 80 cm). As a result, σ was pre-set to 30 to better mitigate the errors.

The GaussianBlur function in the OpenCV library was used to perform Gaussian blur processing to reduce the noise and details in the image. A Gaussian kernel of size 199 was applied to the input image and the variance was set to 30. This process can effectively smooth the image by weighted-averaging each pixel and its surrounding neighborhood pixels, making it Smoother and effectively reducing the raw data of the eye-tracker in the image, including information that is irrelevant to the task.

After this process, the endoscopists’ gaze attention maps were generated, and these maps were then superimposed onto the original endoscopic images to generate the gaze attention heatmaps (Fig. 1), which could be applied to supervise the network attention heatmaps via an attention consistency loss.

Gaze-based attention network

The proposed gaze-based attention network aimed to align network attention with external supervision of the visual attention of endoscopists. This study used EfficientNet_b1, which achieved the best performance in the following experiments Compared with other general models, as the backbone to construct the gaze-based attention network. The architecture of EfficientNet_b1 is shown in Fig. 2. In EfficientNet_b1, the mobile inverted bottleneck convolution (MBConv) block served as a special residual module; it adopted lightweight depth-wise separable convolution and squeeze and extraction (SE) modules to improve the efficiency and performance of the network. The core components of MBConv blocks are divided into two stages: deep convolution and pointwise convolution (Tan & Le, 2019). In the deep convolution stage, the MBConv block independently processes the features of each channel rather than all channels simultaneously, which helps to extract more refined feature information. The output feature map is convolved with a 1 × 1 convolution kernel in the pointwise convolution stage to adjust the number of channels to meet the desired value. Additionally, the SE module in the MBConv block dynamically adjusts the importance between channels, enhancing feature representation. These advantages ensure that the proposed gaze-based attention network has strong feature extraction capabilities.

The proposed gaze-based attention network, as shown in Fig. 3, contained a classification module and an attention consistency module. In the classification module, the input image was first processed through the EfficientNet_b1 backbone to obtain a 16 × 16 × 1,280 feature map. Then, global average pooling (GAP) and flatten applications were applied to obtain a 1 × 1 × 1,280 feature vector. Finally, a fully connected layer of three nodes (representing the three categories of polyps) was used to obtain the final classification result.

In the attention consistency module, weights from the fully connected layer were taken to generate the model feature map. After applying the ReLU function, only the set of weights of the predicted class, which had a 1 × 1,280 shape, was used to perform channel-wise multiplication with the 16 × 16 × 1,280 feature map, and the maps along channels were summed to get a single 16 × 16 map. This map was scaled to the same shape as the gaze attention map and was activated through the sigmoid function to acquire the network attention heatmap. To achieve multi-task learning, this study designed a joint loss function that included classification loss and regression loss to optimize the performance of the model on both classification and regression tasks. In classification tasks, cross-entropy loss (CE) was used to optimize the classification ability of the model to more closely match the true labels.

CE was used to measure the difference between the predicted category (x) and the true category (label) of the model. The CE loss can be formulated as:

(4) $$CE\mathrm{\_}Loss=-\sum _{i=1}^n{y}_i\log (p_i).$$

In this equation, n represents the number of classifications, y_i represents the true label of the i-th class, and p_i represents the predicted label of the i-th class by the model.

In the regression task, the model used gaze attention heatmaps and network attention heatmaps to calculate the mean squared error (MSE) loss function. Mean square error measures the accuracy of model prediction by calculating the square difference between the predicted and actual values of each sample and then averaging the differences among all samples.

Using MSE as the regression loss function to measure the difference between two attention maps, the MSE loss can be formulated as:

(5) $$MSE\mathrm{\_}L\mathrm{o}\mathrm{s}\mathrm{s}={\displaystyle \frac{1}{N}\sum _{i=1}^N\sum _{j=1}^M(A_{model}(i,j)-A_{human}(i,j){)}^2.}$$

In this equation, N and M are the height and width of the image, respectively, while A_model(i,j) and A_human(i,j) are the values of the network attention heatmap and the endoscopists’ eye gaze attention heatmap, respectively, at position (i,j). This loss function represents the difference at each pixel position between the attention heatmap generated by the model and the endoscopists’ eye gaze attention heatmap and calculates the loss by summing up the differences at all positions.

The total loss combines the classification task (CE_Loss) and the gaze prediction task (MSE_Loss). By using weighted combination, this value can provide insight into the quality of the classification and show that the predicted heat map of the network is close to the true value. The total loss can be formulated as:

(6) $$LOSS=CE\mathrm{\_}L\mathrm{o}\mathrm{s}\mathrm{s}+MSE\mathrm{\_}L\mathrm{o}\mathrm{s}\mathrm{s}.$$

Dataset construction

The colorectal polyp gaze dataset was constructed using Narrow Band Imaging (NBI) images of colorectal polyps and the corresponding gaze attention images (i.e., gaze attention maps and gaze attention heatmaps). All data collection and annotation processes were carried out by following the tenets of the Declaration of Helsinki.

A retrospective search was conducted of the NBI colonoscopy observation data at Xiangyang Central Hospital from January 1, 2023 to March 10, 2024. The criterion for inclusion was that patients with colorectal polyps must also have had corresponding pathology reports (the “gold standard” for diagnosis). According to this criterion, 585 NBI images with colorectal polyps were collected from the records of 87 patients. A senior endoscopist classified the 585 NBI images into three categories based on the NICE classification method. During this classification process, the endoscopist’s eye movement information was recorded and used to generate the gaze attention images. A patient-level data splitting was then performed to develop and validate the proposed methods. The selected patients and their corresponding images were randomly split into three sets, of which 60% was used for training, 20% was used for validation, and the remaining 20% was used for testing. The training and validation sets included the original image and the gaze attention images generated from eye movement. In the test set, only the original NBI images were included to test the actual performance of the trained models. The detailed information on the dataset is shown in Table 2.

Experimental setup

A CNN is a complex model with many parameters and requires a large amount of data to obtain more applicable results. With the limited amount of annotated training data, image augmentation was adopted to increase the diversity of training data and improve the generalization ability of deep learning models. Before training, the images were preprocessed and enhanced by randomly applying various rotational transformations, randomly flipping left and right, up and down.

To ensure efficient training and performance optimization of the model in the colorectal polyp classification tasks, this study designed the key parameters of the experiment to meet the special requirements of medical image classification tasks. The backbone of the classification model was pre-trained using ImageNet and then fine-tuned on the experiment’s dataset.

The optimizer used for this study was Adam (Adaptive Moment Estimation), which is an adaptive optimization algorithm that combines the advantages of momentum (a method to speed up training) and RMSProp. Adam can dynamically adjust the learning rate of each parameter, quickly approaching the optimal solution with a larger step size in the early stages of training and finely optimizing the parameters with smaller steps in the later stages of training. This feature is particularly suitable for medical image classification tasks with high-dimensional feature spaces and limited data samples. In addition, the Adam optimizer is less sensitive to hyperparameter settings and can provide a more stable convergence performance during model training. In order to further improve optimization efficiency, this study set the initial learning rate to 0.001, which is based on commonly used empirical values in medical classification tasks, and the optimizer’s best performance was verified through multiple experiments (Zou et al., 2021).

The learning rate scheduling strategy adopted cosine annealing warm restarts, which can dynamically adjust the learning rate at different stages of training to balance the requirements of fast convergence and meticulous optimization. Specifically, this study used the CosineAnnealingWarmRestarts scheduler in PyTorch (Loshchilov & Hutter, 2016), with an initial annealing period set to 10 epochs (T₀ = 10) and subsequent periods increasing exponentially (T_mult = 2). In each cycle, the learning rate gradually decreases from high to low according to the cosine function, thereby promoting the model to quickly jump out of local optima, preventing premature entry into local minima, and improving the robustness of the model. At the end of the cycle, a restart mechanism is used to restore the higher learning rate and explore new solution spaces.

For data processing, the training batch size was set to 8 and the validation batch size was set to 16 in order to balance GPU memory and training efficiency. The training batch size was as large as possible to improve computational efficiency. Due to the limitations of device memory, this selected training batch size could avoid memory overflow and provide more frequent gradient updates to accelerate optimization. During the verification phase, there is no requirement to calculate gradients, therefore larger batch sizes could be used to improve verification efficiency. Data augmentation (random flipping and rotation) was simultaneously used to increase sample diversity and reduce the risk of overfitting. This study set the maximum training epoch to 150 and dynamically adjusted the model preservation strategy based on the performance of the validation set. After each round of training, the classification accuracy of the model on the validation set was evaluated. If the validation accuracy of the current model was better than the historical best value, the weight parameters of the model were saved to ensure the optimal performance of the final model on the validation set. All experiments were run on a 12 GB NVIDIA RTX 4070Ti GPU, which has high-performance computing capabilities and can efficiently support the training needs of high-resolution medical images.

Evaluation metrics

Quantitative evaluation was conducted by comparing the model’s best test accuracy (Acc), precision (Pre), recall (Rec), F1-score (F1 Score), Matthews correlation coefficient (MCC), and Cohen’s kappa (Kappa). The definitions of these evaluation metrics are as follows:

(7) $$Acc={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

(8) $$Pre={\displaystyle \frac{T\mathrm{P}}{TP+FP}}$$

(9) $$Rec={\displaystyle \frac{TP}{TP+FN}}$$

(10) $$F1Score=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(11) $$\mathrm{M}\mathrm{C}\mathrm{C}={\displaystyle \frac{TP\cdot TN-FP\cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}}$$

(12) $$Kappa={\displaystyle \frac{p_0-p_e}{1-p_e}}$$

TP, FP, TN, and FN represent true positive, false positive, true negative, and false negative, respectively. P₀ represents the observed consistency (i.e., the probability of a correct prediction). P_e represents random consistency (i.e., assuming the probability of consistency when the model makes random predictions). As this is a multi-class classification task, the “one-vs-rest” method was used to calculate the performance parameters of each class, treating the target class as a positive class and the remaining classes as negative classes.