Hyperparameter optimization of convolutional neural networks using particle swarm optimization for diabetic retinopathy detection

Pre-trained CNN architectures

A significant challenge in medical image classification is its dependence on data. Due to the uniqueness of medical imaging, access to medical data for research purposes can sometimes be limited. Nevertheless, overfitting may influence experimental results when training DNN designs, such as CNNs, with inadequate data. This study presents transfer learning as an outcome. DL employs a technique known as transfer learning, which involves the transfer of knowledge from one expert to another. By leveraging the similarities across data, tasks, or models like ImageNet, this technique leverages information from one domain (the source domain) to enhance learning in another domain (the target domain). The point of this study was to show the efficacy pre-trained models (VGG16, DenseNet121, and MobileNetV2) and metaheuristic algorithms can improve CNN models’ hyperparameters for classifying DR fundus images and finding DR early on.

VGG16 model

The Visual Geometry Group (VGG16) model, developed by Simonyan & Zisserman (2014), is a CNN with 16 layers and a total of 138,355,752 parameters. The architecture has 13 convolutional layers and three fully linked layers, all of which employ ReLU activation. The design streamlines previous methodologies by employing many 3 $\times$ 3 filters instead of larger kernels. The VGG16 model applies a sequence of convolutional layers to 224 $\times$ 224 RGB images. Max pooling layers follow these layers, which include progressively increasing numbers of filters (64, 128, 256, and 512). Three completely linked layers, each containing 4,096, 4,096, and 1,000 neurons, compose the network. Despite its high processing cost, this complex design has demonstrated significant efficacy in image recognition tasks, rendering it a widely favored option for diverse applications (Albashish, 2022), such as our investigation on diabetic retinopathy categorization. Figure 3 displays the architecture of the basic VGG16 (Simonyan & Zisserman, 2014).

DenseNet-121 model

DenseNet (Huang et al., 2017) is a convolutional network design that enhances typical CNNs by establishing dense connections between each layer and every other layer in a feed-forward manner. A dense network with L layers has a total of $\frac{L(L+1)}{2}$ direct connections. These connections let each layer receive feature maps from all previous layers and contribute its own feature maps to all subsequent layers. This approach facilitates enhanced information dissemination, the reuse of features, and the optimization of parameters (Huang et al., 2017). DenseNet121 is a specific implementation consisting of four dense blocks, with each block containing 6, 12, 24, and 16 convolution blocks, respectively. Each convolution block consists of two convolution layers, each measuring 1 $\times$ 1 and the other 3 $\times$ 3. The transition layers that separate the dense blocks include a 1 $\times$ 1 convolution layer and a 2 $\times$ 2 average pooling layer. The network begins with a 7 $\times$ 7 convolutional layer and ends with a fully linked layer for classification. The network starts with a 7 $\times$ 7 convolutional layer and concludes with a fully connected classification layer, resulting in a total of 121 layers. The approach has several benefits, such as reduced parameter count, faster training, and the capability to utilize all feature maps in the network for classification decisions (Tan et al., 2018). Figure 4 shows the architecture of the DenseNet121 network.

MobileNet model

Google introduced MobileNet (Howard, 2017), a deep learning framework designed exclusively for mobile devices. The exceptional computational performance and small dimensions of MobileNet set it apart. The MobileNet architecture may attain exceptional accuracy rates with a restricted number of hyperparameters (Khudaier & Radhi, 2024). The design predominantly depends on depth-wise separable convolution. The design has 32 initial convolution layers, followed by 19 bottleneck layers. Figure 5 displays the MobileNetV2 model. The conventional 2D convolution performs simultaneous processing of all input channels to generate a solitary output channel while also convolving across the depth dimension (channel). Conversely, depthwise convolution partitions the input image and filters it into distinct channels, subsequently conducting convolution on each input channel using its corresponding filter channel. After generating the filtered output channels, we combine them to form a stack. Separable depthwise convolution is a process in which the stacked output channels are filtered using a 1 $\times$ 1 convolution, also known as pointwise convolution, to merge them into a single channel.

CNN hyper-parameter optimization using PSO

A CNN’s hyperparameters are critical parameters established before model training, regulating the learning process and influencing performance. Thus, this research seeks to design a method to optimise the pre-trained models’ hyperparameters, i.e., VGG-16, DenseNet121, and MobileNetV2. In particular, the performance of the pre-trained models is marginally dependent on their hyperparameters, which remain a substantial obstacle in building pre-trained models.

This section focuses on tuning hyper-parameters in pre-trained CNN models to categorize DR images. The system utilizes a sophisticated metaheuristic approach known as the PSO algorithm. The primary objective is to identify the critical hyperparameters that significantly impact the performance of pre-trained models such as VGG-16, DenseNet121, and MobileNetV2. The PSO approach is used to find the ideal values for these parameters.

A key part of improving CNN models’ performance is the PSO algorithm, which changes important hyperparameters like the number of filters in convolutional layers, the learning rate, and the dropout rate in fully connected layers. By manipulating the number of filters, PSO enhances the ability of convolutional layers to capture intricate information in images. Additionally, by optimizing the dropout and learning rates, PSO improves the performance of fully connected layers during training, resulting in enhanced accuracy of the outcomes. These improvements boost the efficiency of feature extraction, mitigate overfitting, and facilitate successful learning of the model. Collectively, these changes result in a more robust and accurate model, which is particularly crucial for complex tasks in DR classification. The experimental findings provide substantial enhancements in accuracy and other performance measures, highlighting the critical role of PSO in improving the overall performance of CNN.

The conducted experiment compared VGG-16, DenseNet121, and MobileNetV2 to see how well they worked before and after PSO was added. Given that variations in CNN hyperparameters can lead to notable disparities in outcomes for identical tasks, we aim to ascertain these models’ most optimal hyperparameter configurations. Prior research and empirical evidence informed the selection of these hyperparameters, demonstrating their substantial influence on the model’s performance in tasks involving categorizing medical images. This study aimed to optimize three factors in dense layers: the number of filters, the incidence of dropouts, and the learning rate. Increasing the number of filters in CNNs improves the extraction of features and maintains a balance between model complexity and performance. This ensures that the model can effectively adapt to various datasets and efficiently utilize computing resources. Furthermore, choosing an optimal learning rate is critical because it directly impacts the pace at which training algorithms converge. A learning rate that is too little may lead to delayed convergence, while a rate that is too great may introduce instability. Applying an adequate dropout rate is critical for CNN regularization because it promotes model generalization and mitigates overfitting.

The objective is to utilize the PSO technique to enhance the efficiency and precision of pre-trained models in tasks related to DR categorization. This optimization approach aims to significantly improve the models’ performance, providing essential insights into the most optimal CNN hyper-parameter configurations within this domain.

Algorithm 1 provides a detailed elucidation of the PSO technique and its implementation in the methodology. The function g: ${\mathbb{R}}^k\to \mathbb{R}$ represents an objective function derived from the given method. In this case, $k$ stands for the hyper-parameters and $i$ for the fitness function that checks how well the trained VGG-16, DenseNet121, and MobileNetV2 models can classify things. The study’s objective is to find a solution $\lambda$ that satisfies the condition $g(\lambda )\ge g(\lambda ),\lambda \in \chi$, where $\chi$ represents the whole set of potential hyper-parameter configurations.

PSO employs a swarm of particles to facilitate the evolution process, with each particle representing a potential solution, namely a collection of hyper-parameter values. Each particle inside the swarm is characterized by its position within the search space, and its velocity, which determines its movement, denotes the most optimal location attained by any particle there. In contrast, it represents the best site acquired by the particle individually. This study suggests a PSO technique that is not dependent on the specific architecture of the CNN under optimization, and it could be employed in any suggested CNN model. The proposed PSO approach consists of two essential processes: “Swarm Initialization” and “Swarm Evaluation.

Evaluation of the swarm

During each generation (denoted as gen, where $G_{\mathrm{m}\mathrm{a}\mathrm{x}}$ specifies the maximum number of generations), the velocity values of all particles in the swarm are updated using the Eq. (1). It is worth noting that there is a relationship between specifying a certain number of hyper-parameters and the efficiency of the PSO algorithm, which is expressed as follows:

(4) $$TPSO=S\cdot G({\lambda }_k)\cdot g_{\mathrm{m}\mathrm{a}\mathrm{x}}.$$

Equation (4) expresses the time complexity of the PSO algorithm as a product of three factors. Here, TPSO represents the total computational cost. The term S refers to the number of particles in the swarm, $G({\lambda }_k)$ is the time required to evaluate the fitness function for a given set of $k$ hyperparameters ${\lambda }_k$, and $g_{\mathrm{m}\mathrm{a}\mathrm{x}}$ denotes the maximum number of generations. Since each particle evaluates the fitness function in every generation, the total time increases proportionally with the number of particles and generations. Therefore, this equation provides a simple way to estimate the computational effort required by the PSO algorithm when tuning multiple hyperparameters. This estimation is essential for evaluating the scalability and computational feasibility of PSO in real-world hyperparameter tuning tasks. The general structure of the steps involved in optimizing the hyperparameters of pre-trained CNN models using the PSO algorithm is illustrated in Fig. 6.

• •

  Input database for pre-trained CNN models training: In this stage, the APTOS 2019 dataset to be analyzed and classified by VGG-16, DenseNet121, and MobileNetV2 is selected. It is crucial to ensure that all components of the dataset have the same pixel size, file format, and consistent structure.

• •

  Generate the particle population for the PSO algorithm: A PSO algorithm is initialized with specific numerical parameters such as number of iterations, number of particles, inertia weight, cognitive constant (C1), and social constant (C2). Table 2 defines the PSO parameters that were employed during the experiment.

• •

  Configure the pre-trained CNN models architectures: The VGG-16, DenseNet121, and MobileNetV2 architectures are created using the PSO-optimized hyperparameters (number of filters, dropout, and learning rate) and other parameters specified in Table 2.

• •

  Validation and training of pre-trained CNNs architectures: In this stage, the VGG-16, DenseNet121, and MobileNetV2 models read and process the input databases, collect images for training and validation, and generate classification accuracy rates. These values are returned by the objective function to the PSO algorithm.

• •

  Evaluate the objective function: In this step, the PSO algorithm performs an evaluation of the objective function through the equation below to obtain appropriate parameters: (5) $${\mathbf{x}}_i^t(t+1)={\mathbf{x}}_i^t(t)+{\mathbf{v}}_i^t(t+1).$$ In Eq. (6), the particle’s position is updated by adding the newly computed velocity to its current position, enabling exploration of the search space.

• •

  Update the PSO parameters: At each iteration, each particle in the PSO updates its speed and position based on its personal best position (Pbest) in the search space, and the global best position (Gbest) of the swarm as a whole.

• •

  Repeat the process: In our study, the number of repetitions is considered a criterion for stopping. The process continues until all particles are evaluated and the specified stopping conditions are achieved.

• •

  Select the best solution: In this phase, the particle with the global best position (Gbest) represents the best solution for the VGG-16, DenseNet121, and MobileNetV2 models.

Table 2:

Hyperparameter configuration adopted for pretrained CNN models in the proposed transfer learning framework.

Parameters	Ranges
Input size	$224\times 224\times 3$
Output size	5
Optimizer	Adam
Activation function	Softmax
Number of epochs	40
Activation function for activation layer	ReLU
Dropout	0.4
Loss function	Categorical crossentropy
Batch size	32
Learning rate	0.00001

DOI: 10.7717/peerj-cs.3273/table-2

Through the PSO algorithm, our objective is to enhance the performance and precision of pre-trained models in classifying DR images. Our expectation is that the optimization process will immensely contribute to the efficiency and effectiveness of the model. This will give valuable insights into the optimal setting of CNN hyperparameters in this environment. For transparency and replicability of the experimental setup, we explicitly show what hyperparameters optimized with PSO for each CNN model are employed. They are the number of filters (N), dropout rate (D), and learning rate (L). The optimal hyperparameter values and the corresponding accuracies for VGG-16, DenseNet121, and MobileNetV2 were obtained by the suggested PSO-based tuning process.

Dataset

The Kaggle APTOS 2019 Blindness Detection Competition utilizes the Asia Pacific Tele-Ophthalmology Society APTOS 2019 dataset, which is publicly accessible on Kaggle (https://www.kaggle.com/c/aptos2019-blindness-detection/data, accessed on February 17, 2022). This challenge is to prefer deep learning models to analyze fundus images by themselves to discover early manifestations of DR, especially in remote areas where the tests are complex and take a long time (Taufiqurrahman et al., 2020). The dataset consists of 3,662 retinal images captured from various ophthalmology centres in India (the At Aravind Eye Hospital) using fundus imaging techniques and under various photographic conditions. This set also labels images into five categories: no DR (category 0), mild DR (Category 1), moderate DR (Category 2), severe DR (Category 3), and proliferative DR (Category 4). Table 1 reveals a notable imbalance in the distribution of the categories in the APTOS data set; 49% of the images belong to the non-proliferative DR category, 8% to the proliferative retinopathy category, and 5% to the severe disease category. Additionally, Fig. 7 displays some sample images extracted from the APTOS 2019 challenge data set for detecting blindness.

As shown in Table 3, despite categorizing the data into five classes, there is a significant variation in each class sample, creating an imbalance in the data set. Such malfunction may lead to a high error rate in the classification and a decrease in the overall performance of the models. Data augmentation technology will be one of the possible ways to solve the problem. The extended APTOS dataset was created using traditional data augmentation with the original dataset, horizontal and vertical reflection, and adjustments in brightness level.

Data pre-processing

• •

  Proportional resizing

  Pre-processing was initiated by changing the dimensions of the dataset proportionally to $224\times 224$ pixels and initiated model training for each pre-trained CNN model. This will accelerate model training of every pre-trained CNN model, such as VGG16, DenseNet121, and MobileNetV2.

• •

  Image Normalization

  First of all, in this research, the data was re-measured using the minimum and maximum approach in data normalization, in which all pixel values were converted to the range between 0-1, as shown below: (6) $$N_i=\frac{o-o_{\mathrm{m}\mathrm{i}\mathrm{n}}}{o_{\mathrm{m}\mathrm{a}\mathrm{x}}-o_{\mathrm{m}\mathrm{i}\mathrm{n}}}$$where $N_i$ denotes the normal image, and O indicates the original input image data, while $O_{\mathrm{m}\mathrm{i}\mathrm{n}}$ indicate the lower and upper values of the input image data.

• •

  Data augmentation

  This study found that a significant number of training images is required to enhance the accuracy of CNN deep models. The chosen dataset, despite its abundance of photographs, proved insufficient. We must maximally diversify the training data to enhance the model’s stability. Therefore, we employed the data augmentation approach to increase the quantity and diversity of accessible pictures, improving the prediction accuracy. The APTOS 2019 dataset exhibited an unequal distribution of images among five classifications. Data augmentation enabled us to rebalance the dataset classes, thereby diminishing the prevalence of minority classes. The employed image augmentation techniques include shifting, horizontal, and vertical flipping.

Experimental setup

This study examined optimum hyperparameter selection for CNN models with PSO. The models for hyperparameter optimization were implemented in Python using the Google Colaboratory (CoLab) environment. The Keras API, with TensorFlow as the backend, was utilized to develop and train the DNN architectures. TensorFlow, a prominent open-source DL package, supplied the essential tools for building and assessing the proposed models. The integration of Python and TensorFlow enabled effective development, execution, and evaluation of the methodology. The PSO approach found the optimal hyperparameters for pre-trained CNN models.

All tests were conducted utilizing TensorFlow, the Keras API, and Python programming within Google Colaboratory (CoLab) running on a Microsoft machine with a Core i5 processor. Tesla GPUs were employed in CoLab to conduct testing once the dataset was uploaded to Google Drive for accessibility. The CoLab platform provided a practical and collaborative setting for executing computationally demanding experiments, facilitating effective examination of the model’s performance across many hyperparameter combinations. The APTOS 2019 competition dataset was employed, with an 80% training and 20% testing split (Alyoubi, Abulkhair & Shalash, 2021). This configuration enabled a thorough assessment of our hyperparameter optimization methods and their influence on the performance of CNN models.

The proposed approach was to build a foundation for the transfer learning technique with pre-trained models of bypass neural networks, such as CNN. The super parameters represented in Table 2 were kept using the same settings across all applicable models in the transfer learning process. To maintain input compatibility with such models, the images in APTOS 2019 were resized at a resolution of 224 $\times$ 224 pixels. Another critical parameter is the batch size, which was set to 32. The minimum learning rate, $\mathrm{m}\mathrm{i}{\mathrm{n}}_{\mathrm{l}}\mathrm{r}$, is also set to 0.00001, which helps achieve accurate weight adjustments. Moreover, the number of training epochs was set to 40, an adequate iteration for models to gain effective knowledge generalization from available data (Alnabhan et al., 2022).

By fine-tuning these hyperparameters and leveraging the knowledge embedded in pre-trained CNN architectures through transfer learning, the proposed approach aimed to achieve optimal performance on the DR detection task. The careful selection of hyperparameters, such as batch size, learning rate, and the number of epochs, was crucial in ensuring stable training and maximizing the models’ predictive capabilities.

Evaluation metrics

This work used the benchmark APTOS 2019 dataset (Kassani et al., 2019) to deal with a multiclass (five-class) classification challenge for diabetic retinopathy detection. This work used the benchmark APTOS 2019 dataset for a multiclass (five-class) classification challenge for DR detection. Using the proposed PSO, this study enhanced the performance of pre-trained CNN architectures on this demanding task.

In this work, the revised models were tested with CNN using the following various performance metrics: accuracy (ACC), specificity (SP), sensitivity (SEN), future operating characteristic curve analysis (ROC), and accuracy, recall, and F1-score metrics. All these metrics are extracted from the confusion matrix. The confusion matrix classifies the model predictions into four categories: true negatives (TN), true positives (TP), false negatives (FN), and false positives (FP). TN, in the context of the classification of DR, represents those cases where healthy patients were correctly classified as uninfected or non-proliferative. TP shows patients with DR who have been correctly diagnosed with the disease. FP is used to showcase whereby healthy patients were incorrectly classified as having DR, while FN indicates patients with DR who have been wrongly classified as normal. These scales are calculated using Eqs. (7), (8), (9), (10), and (11), as follows:

(7) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}\times 100$$

(8) $$\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{t}\mathrm{y}=\frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}\times 100$$

(9) $$\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}=\frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}\times 100$$

(10) $$\mathrm{F}1\text{-}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=\frac{2\times (\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}\times 100.$$

Further, Kappa statistics is used to evaluate the performance of the proposed strategy. The Kappa statistic calculates the actual accuracy in comparison with the statistical expectation of accuracy using the formula below:

(11) $$\mathrm{K}\mathrm{a}\mathrm{p}\mathrm{p}\mathrm{a}\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=\frac{(\mathrm{O}\mathrm{b}\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{v}\mathrm{e}\mathrm{d}\mathrm{a}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}-\mathrm{E}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{a}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y})}{(1-\mathrm{E}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{a}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y})}.$$

Evaluating pre-trained CNN models evolution

The results of using PSO hyperparameter optimization on CNN models that have already been trained, such as VGG16, DenseNet121, and MobileNetV2, to classify DR are shown in this section. The main focus here is on how this optimization improves the performance of such models compared to their original, unoptimized versions.

Table 4 shows the results concerning the DR classification performance using three pre-trained CNN models: VGG16, DenseNet121, and MobileNetV2. DenseNet121 achieved the best results, achieving an accuracy of 93.85% on the test set. MobileNetV2 followed with an accuracy of 92.79% on the test set. VGG16 proved to have the worst accuracy at 73.7% on the test set. To further present a more realistic evaluation of the reliability of these models, we also review their Kappa values. These ratings provide insight into the degree of agreement between the model’s predictions and actual outcomes, above what would be expected by chance. In this instance, DenseNet121 excels with a Kappa score of 92.30, demonstrating high agreement and reliability in producing reproducible classifications. MobileNetV2 has a Kappa score 90.99, which is slightly higher than VGG16’s score of 67.1; therefore, MobileNetV2 is more reliable. The accuracy and Kappa score for DenseNet121 showed a powerful link, underscoring the model’s reliability. It does not return excellent accuracy alone, but it makes correct predictions regularly; therefore, it is a reliable option for DR classification.

Table 5 displays the results of the hyperparameter optimization using the PSO technique. The PSO-MobileNetV2 model demonstrated exceptional performance on the test test, with an impressive accuracy rate of 97.4% and a Kappa score of 96.77%. This reflects the substantial improvements attained through the optimization process. Similarly, the PSO-DenseNet121 model showed significant improvements on the test set, with an accuracy of 96.34% and a Kappa score of 95.42%. Even though it represented an improvement, PSO-VGG16 was still behind, with an accuracy rate of 76.62% and a Kappa score of 70.77%. Significant gains in accuracy and Kappa score, particularly for the former MobileNetV2, really bring out the effectiveness of PSO optimization in improving model accuracy and the consistency and dependability of the predictions. These results can be considered proof of the power of PSO to improve the accuracy of models in complex classification problems, such as DR.

Results of the pre-trained models with hyperparameter tuning with PSO

The proposed method, which was described in the previous section, focused on developing an improved model through integrating the PSO algorithm and CNN models. The thought behind this method is that it should optimize the hyperparameters of the CNN model in a predefined range so that the accuracy of DR prediction is maximized and the value of the loss function becomes minimized. The optimization method begins by identifying the hyperparameter search space, which defines the range and restrictions for the hyperparameters that will be tweaked. This initial phase is critical because it describes the parameters within which the PSO algorithm will investigate alternative solutions. In our analysis, we used the parameters shown in Table 2, with 40 epochs and a batch size of 32.

To evaluate the effectiveness of our suggested approach, we conducted two experiments. The first applied three unique pre-trained CNN models (VGG16, DenseNetNet121, and MobileNetV2). Second, by incorporating the PSO with pre-trained models, we aimed to identify the ideal hyperparameter configurations that would maximize the performance of these models in the context of diabetic retinopathy classification. The following sections present and analyze the results obtained from our experiments, demonstrating the efficacy of PSO-driven hyperparameter tuning in enhancing the predictive capabilities of CNN models for this critical medical detection.

The proposed PSO-CNN technique was used to tune the hyperparameters of three pre-trained CNN models: VGG16 (Simonyan & Zisserman, 2014; DenseNet121 Huang et al., 2017; and MobileNetV2 Howard, 2017), The findings are described in Tables 6, 7, and 8. The PSO-MobileNetV2 combination outperformed the MobileNetV2 model in predicting diabetic retinal disease. The hyperparameter values [N, D, l] = [237, 0.3, 0.0004] resulted in the most remarkable accuracy of 97.1%, with ‘N’ representing the number of filters, ‘D’ signifying the dropout rate, and ‘l’ representing the learning rate. Several other combinations, including [459, 0.1, 0.0003], [135, 0.5, 0.0008], and [41, 0.8, 0.0007], also yielded outstanding accuracy of more than 96%. Other groups, [333, 0.1, 0.0003] and [118, 0.5, 0.0008], had much lower accuracies, 77.5% and 67.4%, respectively. The DenseNet121 model effectively predicted the sickness. The hyperparameter values [N, D, L] = [32, 0.6, 0.0009] provided the highest accuracy of 96.1%. In comparison, the setup [90, 0.6, 0.0009] produced a lower accuracy of 64.8%. Finally, when detecting diabetic retinopathy using the VGG16 model, the PSO-VGG16 approach fared somewhat worse than the others. The greatest accuracy was 76.5% for hyperparameter values [N, D, L] = [273, 0.5, 0.0004]. Configurations such as [378, 0.8, 0.0003] and [205, 0.4, 0.0001] also yielded accuracy above 72%. However, the configurations [273, 0.5, 0.0004] and [409, 0.3, 0.0008] resulted in lower accuracies of 48.7% and 50.9%, respectively. These results demonstrate the effectiveness of the PSO-CNN method in optimizing hyperparameters [NF, DR, LR] and achieving high accuracies across the APTOS 2019 dataset. In particular, the MobileNetV2 and DenseNet121 models exhibited superior performance with optimized hyperparameters, while the VGG16 model showed relatively lower accuracies. The findings highlight the importance of hyperparameter tuning and the potential of metaheuristic optimization techniques, such as PSO, in enhancing the predictive capabilities of CNN models for DR classification.

This study employed confusion matrices as a comprehensive performance metric to evaluate the efficacy of hyperparameter tuning on pre-trained CNN architectures for multiclass DR classification. Figures 8A–8C presents the confusion matrices for VGG16, DenseNet121, and MobileNetV2. These matrices, generated from the test dataset, provide a detailed breakdown of the models’ classification performance at various DR severity levels. The confusion matrices represent the models’ prediction accuracy by displaying TP, FP, TN, and FN for each DR class. This comprehensive study enables a more nuanced view of each model’s strengths and shortcomings in differentiating across stages of DR development. By comparing the matrices of the three CNN designs, we can see how model selection and hyperparameter tuning affect classification accuracy at each DR severity level. The confusion matrix for pre-trained CNN models and hyperparameter tuning based on PSO shows the number of samples that have been correctly and incorrectly identified and their actual and predicted labels.

Figure 8B shows the confusion matrix, wherein 361, 298, 361, 314, and 360 samples were correctly classified by the DenseNet121 model as DR0, DR1, DR2, DR3, and DR4, respectively. Also, according to the second row of the confusion matrix in Fig. 8B, the model has misclassified 22, 10, 15, and 11 samples of DR1 as DR0, DR2, DR3, and DR4, respectively. The test dataset was used to test all tuned models with PSO-VGG16, PSO-DenseNet121, and PSO-MobileNetV2. From the confusion matrices, it is clear that optimized models have fewer misclassifications (false negatives and positives) than base models, indicating that the modified models outperform the rest.

The confusion matrix in Fig. 9B shows that the PSO-DenseNet121 has properly identified 175, 154, 195, 183, and 187 samples. PSO-DenseNet121, shown in Fig. 9C, and PSO-MobileNetV2 identified 184, 177, 177, 181, and 185 as DR0, DR1, DR2, DR3, and DR4, respectively. Furthermore, the two rows of the confusion matrix in Fig. 9B show that PSO-DenseNet121 misclassified 12, 1, 3, and 4 samples of DR2 as DR0, DR1, DR3, and DR4, respectively. Figure 9C shows that PSO-MobileNetV2 misclassified 6, 2, 5, and 3 DR2 samples as DR0, DR1, DR3, and DR4, respectively.

Figure 10 illustrates the ROC curves for the five DR classes using pre-trained CNN models (VGG16, DenseNet121, and MobileNetV2). It is evident that the DenseNet121 and MobileNetV2 models outperform the VGG16 model across various DR severity grades. Specifically, in Figs. 10B and 10C DenseNet121 and MobileNetV2 demonstrate superior performance in identifying moderate and PDR grades, indicated by larger ROC curve areas.

The performance in identifying the no-DR grade is also notably high for both models, followed by their performance in detecting the severe grade. The mild grade shows the lowest ROC curve area. The suggested DenseNet121 scored 97%, 91%, 99%, 95%, and 99% for no-DR, mild, moderate, severe, and PDR, respectively. At the same time, the MobileNet model achieved 95%, 90%, 99%, 95%, and 99% for NO DR, mild, moderate, severe, and PDR, respectively.

Furthermore, the ROC curves of PSO-DenseNet121 and PSO-MobileNetV2, shown in Figs. 11B and 11C, appear to outperform PSO-VGG16, with near-perfect classification across all severity levels. PSO-VGG16, while still performing well, has more significant variability in its ability to categorize different severity levels. Figure 11B shows that the suggested PSO-DenseNet121 scored 1.00%, 0.98%, 1.00%, 1.00%, and 1.00% for no-DR, mild, moderate, severe, and PDR, respectively. Figure 11C shows that the PSO-MobileNet model obtained 0.99%, 1.00%, 1.00%, 1.00%, and 1.00% for no-DR, mild, moderate, severe, and PDR, respectively.

Analysis of model results based on GA and PSO

In this part, we compare optimized models with GA and PSO. These comparisons give useful information about the efficiency of the two optimization strategies in tweaking the model hyperparameters. Table 9 shows the performance metrics for both models.

Table 9 VGG-16, DenseNet-121 and MobileNetV2 for the DR classification on the APTOS 2019 dataset using accuracy, qualitative accuracy, recall, F1-score and Kappa statistic. This means that, among the compared models, VGG16 has a good accuracy of 73.7% but has yet to achieve the required levels for qualitative accuracy, recall, F1-score, and Kappa statistic; hence, specific limitations exist in correctly classifying disease stages. On the one hand, the best performance for all the metrics was performed by DenseNet121 and MobileNetV2; really, it elevates them to the position of the best option for the given task.

Therefore, the proposed model performed better than previous methods of transfer learning on all the assessment scales, indicating its importance and potential for DR classification. During the hyperparameter optimization using the GA algorithm, the DenseNet-121 architecture yielded an accuracy of 94.45%, along with high values in qualitative accuracy, recall, F1-score and Kappa statistic. These results reflect the high capability of the model in correctly classifying DR severity scores, making it a promising tool for early detection and diagnosis. The model did even more superbly when PSO was combined with MobileNetV2, yielding an accuracy of 97.41% with the best balance of qualitative accuracy, recall, and F1-score. This would show the model’s efficiency in identifying cases of Dr and enhancing the precision of clinical diagnosis.

Notably, our findings demonstrate the efficacy of both GA and PSO for hyperparameter optimization in pre-trained CNN models. The PSO-MobileNetV2 model achieved an outstanding accuracy of 97.41%, showcasing minimal performance discrepancies across various transfer learning models. This exceptional result underscores the potential of these methods to aid medical professionals in detecting and categorizing. The model’s high accuracy and recall scores confirm its reliability and efficacy in distinguishing between DR grades (no DR, mild, moderate, severe, and proliferative DR). Furthermore, the PSO-MobileNetV2 model demonstrated a remarkably low False Negative Rate (FNR) of 2.58%. This low FNR indicates that the proposed model is highly accurate in identifying positive cases of DR, thereby minimizing the number of missed diagnoses. Additionally, this research chose the metrics for their clinical utility and statistical reliability in DR screening. Recall is also extremely important in this context, since it accounts for how well the model gets true DR cases classified as positive, avoiding delayed treatment due to false negatives. Precision, on the flip side, is what ensures that positive cases found are actual positive cases, preventing false alarms that cause anxiety or expensive follow-up treatment. F1-score, being the harmonic mean of precision and recall, is a balanced measure that is optimal for dealing with class-imbalanced datasets such as APTOS 2019. Finally, the Kappa measure adjusts for agreement by chance, providing a better measure of classifier consistency, critical in high-risk clinical decision-making.

Comparison with existing methods

According to the experimental results, the proposed hyperparameter-optimization strategies (such as GA and PSO) and pre-trained CNN models beat other cutting-edge models in the DR fundus image classification task. The experimental results show that adopting hyperparameter optimization techniques for CNN models to medical image classification tasks increases classification performance.

To highlight the efficacy of our proposed hyperparameter-optimization algorithms, we compared our results on the DR fundus images benchmark dataset with the most recent relevant research. Table 10 presents a comparative analysis of our optimized CNN models against current state-of-the-art studies. The findings indicate that our approach, which leverages GA and PSO for hyperparameter tuning, significantly outperforms existing pre-trained models for DR detection. When applied to the APTOS 2019 dataset, our model achieved exceptional performance across two key metrics: accuracy of 97.41% and Cohen’s Kappa of 96.77%. These results represent a substantial improvement over the previous method. The consistently high values across all metrics underscore the robustness of our approach. Notably, the impressive kappa score of 0.9677 indicates excellent inter-rater reliability, further validating the effectiveness of our model in DR classification. For example, in Bodapati et al. (2020). One of the primary reasons for these superior results is the adaptation of the transfer learning models with the appropriate hyperparameters; additionally, hyperparameter tuning was incorporated using GA and PSO parameters, increasing the ability of the proposed pre-trained CNNs to discriminate between various classes in the DR fundus images dataset. Compared to the VGG16 and VGG19 models given in Kale & Sharma (2023), Sharma et al. (2021), which attained accuracies of 87.31%, the suggested model outperforms them. Similarly, the proposed model outperforms the accuracy reported by Shaban et al. (2020) (89%) and Lahmar & Idri (2022) (93.09%) for the APTOS 2019 dataset. Compared to the recent work by Taufiqurrahman et al. (2020), our hyperparameter tuning utilizing GA and PSO for CNNs performed better. They created a generic MobileNetV2 and then refined the model by merging it with an SVM classifier to fine-tune its parameters. Their classification accuracy for the fundus image dataset was 85%. We discover that the suggested strategy is preferable owing to appropriate hyperparameters determined using the GA and PSO optimization methods. Notably, the authors of Kassani et al. (2019) achieved an accuracy of 83.09% for multi-classification on the DR dataset by utilizing the modified Xception pre-trained model. Similar studies were previously conducted to evaluate the categorization of the APTOS 2019 dataset. Based on the findings, the suggested hybrid CNN models can distinguish five different classes in fundus images. Both hybrid CNN with ResNet and hybrid CNN with DenseNet had validation accuracy of around 93.18% and 96.22%, respectively. These results beat the VGG16 and Xception results in Bodapati et al. (2020) by around 16%.

In all comparisons, our hyper-tuned CNNs utilizing metaheuristic optimization consistently outperformed conventional approaches regarding accuracy, precision, and the recall across various testing scenarios. These findings provide strong empirical support for the critical role that GA and PSO play in fine-tuning hyperparameters of pre-trained CNNs. The optimization achieved through GA and PSO significantly enhances classification efficiency and accuracy, underscoring their essential contribution to improving the performance of medical image classification models.

Our GA and PSO-based hyperparameter optimization strategy has proven highly effective in fine-tuning CNN models for DR detection. This approach enhances the model’s performance and demonstrates the potential for evolutionary algorithms in optimizing deep learning architectures for medical image analysis. Figure 12 illustrates the comparative classification accuracy of pretrained CNN models optimized using GA and PSO algorithms.

Although the models trained using PSO demonstrated encouraging outcomes, it is crucial to recognize certain constraints:

• The limited size of the dataset may hinder the model’s capacity to apply its knowledge to new, unknown data successfully. This emphasizes the need to use more enormous datasets or transfer learning methods to improve performance.

• One might use resampling techniques and adjust class weights to enhance the model’s quality and address data imbalances.

• PSO requires many resources and incurs significant computational charges, resulting in longer training durations and higher prices.

Utilizing contemporary tools and state-of-the-art computer technologies could overcome these obstacles, leading to increased production and reduced expenses.

Ethics and Privacy

This study strictly adheres to established ethical standards for medical imaging research. All datasets employed are publicly available and fully anonymized to ensure patient confidentiality and protect privacy. No personally identifiable information was accessed or processed during any stage of the experiments. Furthermore, the research complies with all relevant institutional and international guidelines governing the ethical use of medical data. Potential biases or ethical concerns related to data collection and usage were carefully identified and appropriately addressed.

Relevance from the medical perspective

The outcomes of this research exhibit evident clinical significance in illustrating how more sophisticated DL models can enhance the early diagnosis and grading of DR in regular healthcare practice. Through PSO optimization of CNN hyperparameters, the suggested framework achieves more precise and dependable diagnostic outcomes, thus facilitating ophthalmologists to make better-informed clinical decisions. This strategy has particular utility in settings with limited resources, where access to specialized clinicians could be limited, since it allows for quicker screening and prompt interventions to avert vision loss. The improved model performances constitute an important advancement toward the adoption of CAD tools in daily clinical routines, with the long-term objective of enhancing patient outcomes while alleviating the pressure on healthcare systems.