RetinoNet: an EfficientNet and feature pyramid network-based framework for accurate diabetic retinopathy classification

Data pre-processing pipeline

The data pre-processing pipeline has been used to improve the quality of raw retinal images suitable for feature extraction during DR detection. The pipeline also consists of various imaging processing steps to denoise and contrast-enhance the images and prepare it so that the features can be accurately and consistently extracted. This section lists all the steps and their wares in sequentially to assist to neutralize images and comprehensive to numerical analysis. Statistical tests have been used to ensure no bias was introduced during enhancement, maintaining consistency and reliability (Özbay, 2023). The retinal images have been converted into grayscale to allow for further structural analysis and to extract features. To convert to grayscale, it uses the following formula:

(1) $${\mathrm{I}}_{\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{y}}\left(\mathrm{x},\mathrm{y}\right)=0.2989\times \mathrm{R}(\mathrm{x},\mathrm{y})+0.5870\times \mathrm{G}(\mathrm{x},\mathrm{y})+0.1140\times \mathrm{B}(\mathrm{x},\mathrm{y}).$$

This reduces the image for DR detection providing simpler global representation while preserving structural features. GrabCut algorithm was applied for the segmentation of the retina to achieve the region of interest (ROI):

GrabCut energy function:

(2) $$\mathit{E}\left(\mathrm{x}\right)=\sum _{\mathrm{i}\in \mathrm{f}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{d}}-\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{P}({\mathrm{x}}_{\mathrm{i}}|{\mathrm{\theta }}_{\mathrm{F}})+\sum _{\mathrm{i}\in \mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{d}}-\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{P}({\mathrm{x}}_{\mathrm{i}}|{\mathrm{\theta }}_{\mathrm{B}}).$$

The central region of the greyscale fundus image (the retina that is the most interesting for the clinical fundus analysis) was specified as a seed of the foreground in order to have enough regions of interest to segment the fundus image, and the remaining part was set as a seed of the background seed, given that most of the fundus images have a similar anatomy where there is likely to be background. This procedure ensures that the retina is isolated from the background and results in an in-focus image, denoted as ${\mathrm{I}}_{\mathrm{m}\mathrm{a}\mathrm{s}\mathrm{k}\mathrm{e}\mathrm{d}}$. Next step is applying Contrast Limited Adaptive Histogram Equalization (CLAHE) to increase local contrast and enhance the DR features like; microaneurysms.

The CLAHE uses the formula:

(3) $$\mathrm{C}\mathrm{L}\mathrm{A}\mathrm{H}\mathrm{E}\left({\mathrm{I}}_{\mathrm{m}\mathrm{a}\mathrm{s}\mathrm{k}\mathrm{e}\mathrm{d}}\left(\mathrm{x},\mathrm{y}\right)\right)={\displaystyle \frac{\left({\mathrm{I}}_{\mathrm{m}\mathrm{a}\mathrm{s}\mathrm{k}\mathrm{e}\mathrm{d}}\left(\mathrm{x},\mathrm{y}\right)\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(\mathrm{I}\right)}{max\left(\mathrm{I}\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(\mathrm{I}\right)}\times 255.}$$

Non-local means denoising method have been applied to remove noise thereby preserving important details, such as blood vessels and lesions. The formula used for de-noising is

(4) $${\mathrm{I}}_{\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{d}}\left(\mathrm{x},\mathrm{y}\right)=\sum _{\mathrm{p}\in \mathrm{p}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}\left(\mathrm{x},\mathrm{y}\right)}\mathrm{w}\left(\mathrm{p}\right)\cdot {\mathrm{I}}_{\mathrm{C}\mathrm{L}\mathrm{A}\mathrm{H}\mathrm{E}}\left(\mathrm{p}\right).$$

This results in denoised image $I_{denoised}$, which retains essential DR features, and cubic interpolation is used to upscale the resolution for better visualization of DR features.

The upscaling formula is as follows:

(5) $${\mathrm{I}}_{\mathrm{n}\mathrm{e}\mathrm{w}}\left(\mathrm{x},\mathrm{y}\right)=\sum _{\mathrm{i}=-1}^2\sum _{\mathrm{j}=-1}^2\mathrm{I}\left(\mathrm{x}+\mathrm{i},\mathrm{y}+\mathrm{j}\right)\cdot \mathrm{w}\left(\mathrm{i}\right)\cdot \mathrm{w}\left(\mathrm{j}\right).$$

The up scaled image ${\mathit{I}}_{\mathit{u}\mathit{p}\mathit{s}\mathit{c}\mathit{a}\mathit{l}\mathit{e}\mathit{d}}$provides greater clarity for detecting subtle abnormalities. Finally, CLAHE is reapplied to the up-scaled image to further enhance the contrast using the formula:

(6) $$\mathrm{C}\mathrm{L}\mathrm{A}\mathrm{H}\mathrm{E}\left({\mathrm{I}}_{\mathrm{u}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e}\mathrm{d}}\left(\mathrm{x},\mathrm{y}\right)\right)={\displaystyle \frac{\left({\mathrm{I}}_{\mathrm{u}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e}\mathrm{d}}\left(\mathrm{x},\mathrm{y}\right)\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(\mathrm{I}\right)}{max\left(\mathrm{I}\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(\mathrm{I}\right)}\times 255.}$$

This generates the fully processed image, ${\mathit{I}}_{\mathit{p}\mathit{r}\mathit{e}\mathit{p}\mathit{r}\mathit{o}\mathit{c}\mathit{e}\mathit{s}\mathit{s}\mathit{e}\mathit{d}}$, ready for feature extraction.

The pre-processed retinal images $I_{preprocessed}$ are processed through the backbone of EfficientNet-B0 to model features. This process extracts $F_{low}$, $F_{mid}$, and $F_{high}$ features, which are essential for identifying various DR abnormalities, including microaneurysms, hemorrhages, and blood vessel changes (Farag, Fouad & Abdel-Hamid, 2022).

This process can be represented as:

(7) $$F_{low},F_{mid},F_{high}=fEfficientNetBo(I_{preprocessed}),$$where $fEfficientNetBo$ represents the EfficientNet feature extraction function. FPN upscale and fuses these features to produce a comprehensive multi-scale feature map, enabling the detection of both small and large retinal abnormalities. $F_{low}$, $F_{mid}$, and $F_{high}$ are fused to generate the final feature map:

(8) $$F_{P5}=Conv(F_{high})$$

(9) $$F_{P4}=Conv(F_{mid})+Upsample\left(F_{P5}\right)$$

(10) $$F_{P3}=Conv(F_{low})+Upsample\left(F_{P4}\right).$$

The multi-scale feature map $F_{final}$ is optimized for DR severity detection through a combination of Focal Loss for classification and Smooth L1 Loss for localization. Focal Loss, defined as:

(11) $$FL(p_t)=-{\alpha }_t{\left(1-P_t\right)}^{\gamma }\mathrm{l}\mathrm{o}\mathrm{g}\left(P_t\right),$$addresses the class imbalance inherent in DR detection, while Smooth L1 Loss:

(12) $$SmoothL1\left(x\right)=\{\begin{array}{ll}0.5x^2\hfill & if\left|\text{x}\right|\text{<}1\hfill \\ \left|\text{x}\right|-0.5\hfill & \text{otherwise}\hfill \end{array},$$is used for precise localization of retinal features such as lesions. The total loss function is:

(13) $$L_{total}=L_{classification}+L_{localization},$$guaranteeing accurate classification and precise localization in detecting DR severity stages. The output of RetinoNet is a multi-scale feature map which further reduced in dimensionality using GAP. This feed-forward network is then flattened from the reduced feature map and passed through a dense layer to combine features. It then performs Softmax activation to classify the severity of DR into its categories (Aswini & Vijayaraghavan, 2022; AbdelMaksoud, Barakat & Elmogy, 2022; Bilal et al., 2022).

Figure 3 shows the RetinoNet architecture used for DR detection, featuring convolutional (Conv2D), activation, batch normalization, pooling, and fully connected (Dense) layers. The diagram details the model’s components and the transformations applied to input images. The model starts with input pre-processing (rescaling, normalization), followed by Conv2D and pooling operations for hierarchical feature extraction. Batch normalization, dropout, and feature concatenation are employed to stabilize training and improve generalizability.

Table 1 presents a summary of the essential hyper-parameters of the RetinoNet model. These hyper-parameters are critical for the replication of the model for DR detection. The parameters encompass specifications such as the model architecture, input dimensions, feature extraction settings, optimization strategy, and augmentation techniques, which are instrumental in achieving high accuracy and generalization in model performance (Oh et al., 2021; Singh & Dobhal, 2024).

Visualization of the image enhancement

Figure 4 shows the stages of the enhancements which are obtained for an image. It shows a sample of the processing and segmentation stages used in this study for DR detection: the processes applied over an original image that consider important features of the retina. The pre-processed image is converted to grayscale, with contrast adjustment using contrast-limited adaptive histogram equalization (CLAHE), followed by non-local means denoising to avoid over-smoothing of detailed information. The up-scaled image, which is cubic interpolated, offers a higher resolution and finer details in the images, such as a finer view of the small blood vessels (Han et al., 2022; Huang et al., 2022). In addition, the contrast-enhanced image provides significant contrast to discern microaneurysms and haemorrhages, which are essential for performing segmentation of certain retinal features and for accurate classification. This standardized pipeline applies to all images with consistency and ensures no bias in image enhancement while respecting the variability in DR severity. Validation of image quality has been performed in the content of research using PSNR, SSIM, and MSE, and analysis of variance was conducted on all stages of DR to ensure consistency and unbiased enhancement, thus allowing for reliable segmentation and detection.

ANOVA for DR detection

A one-way ANOVA test was used to determine whether there were statistically significant differences between the means of three or more independent groups. Table 2 shows randomly selected values of metrics used in evaluating the enhanced image from various DR classes namely class 0: No DR, Class 1: Mild DR, Class 2: Moderate DR, Class 3: Severe DR. The abbreviation of each metrics is PSNR, SSIM, and MSE. These metrics are essential in the assessment of image performance. The metrics PSNR is used to assess the signal-to-noise ratio, SSIM measures structural similarity between the original and enhanced images, and MSE quantifies pixel-level error. The objective of performing ANOVA on these random sample was to statistically determine whether or not the output of the enhancement pipeline is unbiased and uniform across all DR severity levels. The ANOVA evaluation of these measures confirms the fairness and consistency of the process, ensuring uniform image quality improvement across all severity classes without any bias towards specific classes. This guarantees the accuracy and fairness of the process in a real-time implementation.

The steps involved to perform ANOVA are given as follows:

Step 1: Define the hypotheses:

• Null hypothesis (H₀): There are no significant differences in PSNR, SSIM, or MSE across DR severity levels.

• Alternative hypothesis (H₁): There are significant differences in PSNR, SSIM, or MSE across DR severity levels.

Step 2: Calculate the F-statistic and p-value

In statistical analysis of variance (ANOVA), the F-statistic compares the variance within each group to the variance between the groups (PSNR, SSIM, and MSE across DR severity levels). The formula is

(14) $$F={\displaystyle \frac{VarianceBetweenGroups(MSB)}{VarianceWithinGroups(MSW}.}$$

• Between-Group Variance (SSB): Measures the variability in PSNR, SSIM, and MSE across different DR severity levels (e.g., comparing Class 0 to Class 3).

• Within-Group Variance (SSW): Measures the variability within each DR class (e.g., how PSNR varies within Class 1).

Step 3: Accept or reject the hypothesis

The chosen significance level is generally 0.05 which is compare the p-value.

• If p-value ≤0.05, reject the null hypothesis (H₀) and conclude that there are significant differences in PSNR, SSIM or MSE across DR severity levels.

• If the p-value is larger than 0.05, it fails to reject the null hypothesis (H₀), concluding that no significant difference exists in PSNR, SSIM, or MSE among the different DR levels

Table 3 shows the results of the ANOVA test, which show that there is no statistically significant difference in image quality metrics across different levels of DR severity. This shows that the method is very reliable. The provided information and statistical analysis provide strong proof that the suggested image enhancement method is effective and fair for all stages of DR.

Statistical and visual analysis of image quality enhancement in DR detection

The histograms and density plots for the distribution of image quality metrics considered in this study are shown in Fig. 5 with detailed interpretations. These plots provide some insight into the effectiveness and consistency of different pre-processing techniques. The minimal variations in the values of PSNR (19.925 to 20.075), SSIM (0.5425 to 0.5600), and MSE (84.2 to 85.8) reflect that the resultant image quality appears quite similar for all samples used for experimentation. We can also confidently ascertain this consistency from each of the values that tend to cluster around some points: PSNR = 20, SSIM = 0.55, and MSE = 85. This type of uniformity is important to ensure that the pre-processing methods do not introduce any bias or differential enhancement across various DR severity levels (Khan et al., 2021).

Assessment metrics used in RetinoNet

The standard assessment metrics used in evaluating the RetinoNet model is discussed in detail

(15) $$Precision:TP/(TP+FP).$$

Precision measures the accuracy of positive predictions, indicating how often DR-positive cases are correctly identified without false alarms.

(16) $$Recall(Sensitivity):TP/(TP+FN).$$

Recall measures the model’s ability to detect all positive cases, ensuring that the model identifies most patients with DR correctly.

(17) $$F1\text{-}score:2\times (Precision\times Recall)/(Precision+Recall).$$

The F1-score balances both precision and recall, especially useful in DR detection where there may be an imbalance between positive and negative cases.

(18) $$Specificity:TN/(TN+FP).$$

Specificity measures how well the model identifies negative cases, ensuring that healthy patients are not misclassified as having DR.

(19) $$Accuracy:(TP+TN)/(TP+TN+FP+FN).$$

Accuracy measures the overall correctness of the model, providing an overall sense of the model’s effectiveness in detecting DR.

The Fig. 6 shows the comparatively strong performance of RetinoNet on precision, recall, F1-score, specificity and accuracy metrics for DR detection on the Messidor and APTOS datasets. The relatively high values for both datasets denote that the estimation model performs close to its upper bound, which indicates strong generalization ability. In addition, performance metrics, such as accuracy and specificity, do not vary significantly, which shows the reliability and consistency of the model for DR detection across classes. This clearly proves that RetinoNet can provide reliable results across all severity levels of DR (Sungheetha, 2021; Gadekallu et al., 2020).

Grad-CAM for lesion interpretation

Grad-CAM is used in this study to highlight the discriminative parts in fundus images that are vital for classification (Bilal et al., 2023). This enhances clinical interpretability and enables validation of the model for targeting of pathologically relevant features. Therefore, it supports clinical trust in model prediction. The DR-affected sample for each class are presented in Fig. 7.

Comparative analysis across model

Dealing with optimizations, including model-type selections, tuning hyper parameters, and creating a fusion of technology in every stage of model creation, this study systematically compares different perspectives of the model. Therefore, a performance comparison was performed to compare the proposed model with other simple models for DR detection. The comparison starts with the evaluation of metrics such as accuracy, precision, recall, and other relevant metrics that are used in evaluating a common DR detection model. The performance comparisons between the various models presented in Fig. 8 indicate the strengths and weaknesses of the models in terms of the performance assessment criteria. The performance comparison is done on both Messidor and APTOS dataset. Based on the comparative analysis, EfficientNet-B0 has been selected as the backbone model for the proposed system due to its superior performance over other baseline architectures. Subsequently, to enhance its feature extraction capabilities, the model was fused with FPN, resulting in RetinoNet. Therefore, the use of EfficientNet-B0 and its fusion with FPN is justified based on the empirical performance comparison. The proposed model achieved a relatively better performance compared to other transfer learning models, with marked improvements in accuracy, precision, and specificity, which are clearly visible in the graph. RetinoNet achieves 96.8% accuracy and 94.8% recall.

Table 4 presents the breakdown of important metrics like precision, recall, specificity, accuracy, and F1-score, allowing for a more detailed numerical analysis. By using both visual and numerical data, this approach enhances clarity and also helps make more informed decisions. Therefore, it enables researchers to identify the advantages and disadvantages of each model, thereby enhancing real-world DR diagnosis and ensuring consistent performance under diverse conditions.

In healthcare, obtaining a correct diagnosis is crucial. Comparing various models enhances comprehension of their own strengths and weaknesses. Prioritizing a model with elevated sensitivity helps diminish false negatives, which is especially vital in the early identification of DR. Therefore, by analyzing model outcomes comparatively, discernible patterns emerge that facilitate algorithm modification and highlight areas needing enhancement (Kaur & Mittal, 2018; Randive, Senapati & Rahulkar, 2019). This procedure facilitates the identification of optimal models grounded in empirical evidence. These efforts not only promote automated DR detection but also enable the conversion of research into applicable clinical instruments, ultimately improving patient care. By collaborating, researchers, physicians, and developers can produce advanced diagnostic tools that meet the needs of practical healthcare situations (Pandey & Sharma, 2018; Gadekallu et al., 2023).

Table 5 helps in understanding the proposed model in comparison with SOTA features such as model technology, optimization type, and comparison insight which indicates the manner in which RetinoNet is wise than other models. The comparison proves that the model RetinoNet, with the introduction of a chain of pre-processing, EfficentNetBo with FPN along with adaptive pipeline, attains promising results with 96.8% accuracy on Messidor and 91.6% accuracy in validation dataset i.e., APTOS, which ensures enhancing both speed and accuracy. There are other models that deal with optimizing in areas like feature selection, segmentation, or hybrid architectures in which our model focuses on optimization techniques chosen to enhance the speed and accuracy (Hu et al., 2025; Wang et al., 2025; Dao, Trinh & Nguyen, 2023). This systematic comparison demonstrated the effectiveness of RetinoNet for DR detection.

Figure 9 presents a bar chart comparing various metrics used in literature survey with the proposed models, i.e., RetinoNet model, for DR detection. This level of performance suggests that the proposed model could be a valuable tool in the field of automated DR detection, potentially contributing to improved screening and diagnosis processes in clinical settings.