Hybrid deep learning approach for brain tumor classification using EfficientNetB0 and novel quantum genetic algorithm

CNN-based model

In our study, a traditional convolutional neural network (CNN) model was developed for brain tumor classification. The model consists of several layers, including convolutional, max pooling, and fully connected layers, designed to extract features and perform classification tasks. The CNN accepts 128 × 128 pixel RGB brain tumor images as input. It uses two convolutional layers (32 and 64 filters) with ReLU activation, followed by max pooling layers to reduce the image size and retain important features. The extracted features are flattened and passed through a fully connected layer with 128 neurons, followed by a softmax layer for classification into three tumor types (glioma, meningioma, pituitary tumor). To prevent overfitting, dropout with a rate of 50% was applied. The model was trained using the sparse categorical cross-entropy loss function and the Adam optimizer, with early stopping and learning rate reduction techniques employed to enhance performance. The dataset was split 80% for training and 20% for testing, and the model was trained for up to 50 epochs, with early stopping applied to halt training when no improvement was observed (Pereira et al., 2016).

EfficientNetB0 based model

In this study, the EfficientNetB0 model was utilized with transfer learning for brain tumor classification. EfficientNetB0, a deep learning model pre-trained on the ImageNet dataset, was fine-tuned for the specific task of classifying brain tumors. By leveraging pre-trained weights, the training time was reduced, and performance was improved. The model’s pre-trained layers were frozen, except for the final layers, which were modified for the classification task. The output features were flattened and passed through a dense layer with 128 neurons, followed by a dropout layer to prevent overfitting. The final softmax layer classified the images into glioma, meningioma, and pituitary tumor categories. The model was trained using the Adam optimizer and sparse categorical cross-entropy loss function, with early stopping and learning rate reduction applied for optimization. The EfficientNetB0 model demonstrated high accuracy in brain tumor classification (Pereira et al., 2016).

Hybrid model

In this study, a hybrid model combining CNN and EfficientNetB0 was developed to improve brain tumor classification accuracy. The hybrid model leverages CNN’s ability to extract low-level features and EfficientNetB0’s strength in extracting high-level features. The features from both models are concatenated and passed through dense layers for classification. This combination enhances performance by utilizing the strengths of both models. The hybrid model was trained with early stopping and learning rate reduction techniques, preventing overfitting. After training, the model was evaluated on a test set, showing high accuracy and superior performance in brain tumor classification (Pereira et al., 2016).

Fine-tuned hybrid model

In this study, the performance of the hybrid model combining CNN and EfficientNetB0 was further enhanced through the fine-tuning method. Fine-tuning is an optimization process where certain layers of a pre-trained model, such as EfficientNetB0, are retrained to better adapt to a new dataset. During the transfer learning phase, the core layers of the EfficientNetB0 model were frozen and not trained. However, in the fine-tuning phase, the final layers of the model were unlocked and retrained to better fit the new dataset. The last few layers of EfficientNetB0 were adapted specifically for the brain tumor classification task, allowing the model to learn more distinct features and improve its overall performance. During training, only the last layers of the EfficientNetB0 model were retrained while the other layers remained fixed, preserving the general features learned from the large-scale dataset and adding more specific features for the new dataset. The fine-tuned hybrid model, combining the strengths of CNN and EfficientNetB0, offered higher accuracy and generalization ability. The model delivered better results on the test data, and its performance was evaluated through classification reports and confusion matrices. Notably, the fine-tuned hybrid model achieved higher accuracy rates compared to the transfer learning model alone (Pereira et al., 2016).

Quantum genetic algorithm

Quantum genetic algorithm (QGA) is an extension of classical genetic algorithms based on quantum mechanical principles. QGA aims to explore the solution space more effectively by using the concepts of quantum bits (qubits) and superposition. The basic components and working principles of this algorithm are as follows:

1. Beginning:

• The algorithm initializes the initial quantum state matrix $Q\left(t\right)$.

• By observing these situations, the initial population $P\left(t\right)$ is created.

• The population is evaluated and the best individual $B$ is kept.

2. Iterative process:

• The maximum number of generations $\left(t_{max}\right)$ is selected.

• The number of iterations is increased $t=t+1$

• A new population $P\left(t\right)$is created by observing the states of the previous quantum state matrix $Q\left(t-1\right)$).

• The new population $P\left(t\right)$ is evaluated.

• The quantum state matrix $Q\left(t\right)$ is updated.

• The best individual in the new population is kept.

3. Update and result:

QGA aims to improve the overall performance of the population by updating qubits based on the best individual.

When the maximum number of generations is reached or other stopping criteria are met, the algorithm is terminated and the best individual $B$ is given as the result.

The features and advantages of QGA are as follows. Quantum bits (Qubits), unlike classical bits, have superposition states that can take both values at the same time, not just ‘0’ or ‘1’. A qubit state |ψ〉 is expressed as:

(1) $$|\mathrm{\Psi }>=\alpha |\theta >+\beta |1>$$

Here α and β are complex numbers, and ${\alpha }^2$ and ${\beta }^2$ values indicate the probabilities of the qubit being in the ‘0’ and ‘1’ states, respectively. As normalization condition:

(2) $${\left|\alpha \right|}^2+{\left|\beta \right|}^2=1$$

In superposition and parallel computing, the superposition state of qubits allows $2^{\mathrm{m}}$states to be represented simultaneously in m-qubit systems. This enables parallel exploration of the solution space and reduces computational complexity.

Regarding the global search capability, QGA reduces the possibility of getting stuck in local minima when searching for global optimal solutions. This allows the algorithm to find optimal solutions more effectively and quickly. In its effectiveness in feature selection, QGA enables the model to work more efficiently and accurately by identifying the most important features of the data when selecting features.

This is a significant advantage, especially in high-dimensional datasets. The QGA pseudo code is as follows.

In the update process, Qubits are updated to approximate the best individual angularly. The update formula is as follows:

(3) $$\left[\begin{array}{c}{\alpha }_t\\ {\beta }_t\end{array}\right]=\left[\begin{array}{c}cos\left(\mathrm{\Delta }{\theta }_t\right)-sin\left(\mathrm{\Delta }{\theta }_t\right)\\ sin\left(\mathrm{\Delta }{\theta }_t\right)cos\left(\mathrm{\Delta }{\theta }_t\right)\end{array}\right]\left[\begin{array}{c}{\alpha }_{t-1}\\ {\beta }_{t-1}\end{array}\right]$$

Here, the $\theta$ angle is best determined according to the fitness function of the individual (Malossini, Blanzieri & Calarco, 2008). A variant of QGA is given in Fig. 2.

Proposed method

In this study, the EfficientNetB0 model was trained on image data. This model was used to solve the brain tumor classification problem and its accuracy was evaluated. Features were extracted from the trained EfficientNetB0 model. This process represents the features obtained from the last layers of the model. Feature extraction was performed on both training and test data. Feature selection was made using QGA on the features extracted from the EfficientNetB0 model. QGA was used to select the most important features in the data. A new model was created with features selected by QGA. This model was trained only on selected features. The new model trained with the selected features was re-evaluated on the test data. In summary, feature selection with QGA was applied on the features extracted from the EfficientNetB0 model, and it was aimed to make a more optimized classification by training a new model with these features. In this section, the step-by-step processes are given as follows and in Fig. 3. Brain tumor classification is a critical task for medical imaging and early diagnosis, and while many studies have explored deep learning approaches, there is still room for improvement in terms of accuracy and efficiency. This is especially important when dealing with limited and imbalanced datasets, where traditional methods may struggle to generalize well. The motivation behind our study is to enhance the performance of brain tumor classification systems by proposing a novel hybrid approach that combines the deep feature extraction capabilities of the EfficientNetB0 model with the powerful feature selection process of a QGA. This combination addresses the challenges of feature selection, improves the model’s efficiency, and leads to better performance in classification, particularly in real-world clinical settings where accurate and rapid diagnosis is crucial. The unique contribution of this article lies in the integration of EfficientNetB0 with QGA-based feature selection (QGA-FS). No previous research has combined these two advanced techniques to address the brain tumor classification problem. This integration leverages EfficientNetB0’s powerful feature extraction capabilities and QGA’s ability to select the most important features, resulting in a more efficient and accurate model. The proposed method demonstrates superior accuracy compared to existing approaches. In our study, the hybrid model achieved accuracy rates of 98.36% on the Figshare dataset and 98.25% on the MRI dataset, outperforming other deep learning models and feature selection techniques in the literature. The proposed model was validated using two different datasets, emphasizing its generalization capacity across different data types. This differentiates our approach from other studies that focus on a single dataset, making our solution more robust for real-world applications. While other methods use genetic algorithms for feature selection, the use of QGAs in combination with deep learning models for brain tumor classification is innovative. QGA enhances the model’s ability to find optimal features in high-dimensional datasets, reducing computational costs and increasing classification accuracy. These contributions distinguish our article from existing studies, offering both innovation and practical value in improving the performance of brain tumor classification.

Datasets

The brain tumor dataset on the Kaggle platform was chosen due to its widespread use and its inclusion in many recent studies. The dataset provides a rich collection of MRI images and enables performance comparisons of commonly used models in the literature. In addition, the labeling quality, diversity, and easy accessibility of the dataset provide significant advantages for brain tumor classification studies. Although other datasets have been evaluated, this dataset was found to be more suitable due to factors such as being widely tested on different models and being constantly updated with feedback from the community. Therefore, we aimed to compare it with other studies in the literature by focusing on the Kaggle dataset in our study and analyze the success level of the proposed model in a wider reference framework. Two different datasets were used: the Figshare Brain Tumor Dataset and the MRI Brain Tumor Dataset. The Figshare Brain Tumor Dataset consists of images from three different tumor classes (glioma, meningioma, and pituitary tumor). MRI Brain Tumor Dataset consists of MRI images containing various types of tumors.

The first data set used in this study was collected by Tianjin Medical University between 2005 and 2010 and published online by Jun (2017). The Figshare dataset consists of 3064 T1-weighted contrast-enhanced (CE) MRI images of glioma, meningioma and pituitary tumors. These images were obtained in three different planes (coronal, sagittal and axial) from 233 patients. Images are 512 × 512 pixels in size, with each pixel measuring 49 mm × 49 mm. The second data set was obtained from the Kaggle website. This dataset also includes Normal samples as well as glioma, meningioma and pituitary tumors. The Kaggle dataset is basically divided into two parts: training and testing samples. The images were collected from various web sources and published online after verification by a medical expert (Bhuvaji et al., 2020).

Experimental protocol

The preprocessing steps applied to the datasets are as follows: All images were resized to 128 × 128 pixels. Black-and-white images were converted to RGB format and made three-channel. Pixel values were normalized to the range [0,1]. The datasets were split into 80% training and 20% testing. Two different approaches were tested using the EfficientNetB0 model and a CNN-based model: A model pre-trained on ImageNet was fine-tuned on brain tumor images using transfer learning. Feature extraction and classification tasks were optimized using a CNN model trained from scratch. EfficientNetB0 and CNN were combined, where EfficientNetB0 was used for feature extraction and CNN for classification. During the parameter tuning process, the model hyperparameters were manually tested. The early stopping mechanism was used to prevent overfitting by stopping training early if no improvement was observed. Additionally, ReduceLROnPlateau was applied to reduce the learning rate when no improvement in validation loss was detected. The dropout rate used in the model was set to 0.5, which is a commonly accepted value in the literature. For the QGA-FS algorithm, the population size and maximum number of iterations were carefully tuned, and the mutation rate was set to 1%. To assess the robustness and generalization capability of the model, an 80–20 split ratio was used for training and testing. The data were randomly divided into training and testing sets, minimizing the risk of overfitting. Moreover, cross-validation was used to further improve the generalization ability of each model. A fixed random seed was employed in each experiment to ensure reproducibility of the results. The hyperparameters used during model training included a learning rate of 0.001, batch size of 32, epoch count of 50, early stopping (stopping training if no improvement in validation loss occurred for five epochs), and ReduceLROnPlateau (reducing the learning rate if no improvement occurred for five epochs).

Evaluation metrics

The proposed method was evaluated with precision, recall, accuracy, F1 score, Cohen’s Kappa score and Matthews correlation coefficient. Precision is specified in Eq. (4), Precision is specified in Eq. (5), and Accuracy is specified in Eq. (6). In Eq. (7), F1 Score is used as the harmonic average of these metrics. To evaluate the performance of the classification models, Cohen’s Kappa coefficient and Matthews correlation coefficient (MCC) statistical measures are provided in Eqs. (8) and (9). Cohen’s Kappa coefficient is a measure that adjusts for chance agreement when assessing the agreement between two classifiers (Cohen, 1960). It is particularly useful in imbalanced datasets where classification accuracy can be misleading. MCC is a balanced measure used in binary classification problems, evaluating the overall performance of the classifier. It provides reliable results even when there is an imbalance between positive and negative classes (Matthews, 1975).

(4) $$Presicion={\displaystyle \frac{TP}{TP+FP}}$$

(5) $$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(6) $$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(7) $$F1Score=2\ast {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(8) $$k={\displaystyle \frac{p_0-p_e}{1-p_e}}$$

(9) $$MCC={\displaystyle \frac{\left(TP\times TN\right)-\left(FP\times FN\right)}{\sqrt{\left(TP+FP\right)\left(TP+FN\right)\left(TN+FP\right)\left(TN+FN\right)}}}$$

Here, TP represents true positives, FP represents false positives, FN represents false negatives, TN represents true negatives. F1 score is the harmonic mean of precision and sensitivity and is used to evaluate model performance on unbalanced data sets.

Proposed method results

In this study, first, the CNN method was used, as seen in Table 1. This method had precision values of 87%, 92% and 98% in the classification of meningioma, glioma and pituitary tumors, respectively. The overall accuracy rate was calculated as 92.65%. The transfer learning method had precision values of 94%, 98% and 98% in the classification of meningioma, glioma and pituitary tumors, respectively. The overall accuracy rate of this method was calculated as 97.22%. The hybrid model (CNN + EfficientNetB0) had precision values of 97%, 98% and 99% in classifying meningioma, glioma and pituitary tumors, respectively. The overall accuracy rate was calculated as 98.20%. The fine-tuned hybrid model (Fine Tuning Hybrid) had precision values of 50%, 67% and 69% in the classification of meningioma, glioma and pituitary tumors, respectively. The overall accuracy rate was calculated as 66.06%. Finally, the proposed EfficientNetB0 model and the QGA-FS method had precision values of 96%, 100% and 98%, respectively, in the classification of meningioma, glioma and pituitary tumors. The overall accuracy rate was calculated as 98.36%. Both Cohen’s Kappa and MCC scores reveal that the proposed EfficientNetB0 model with QGA-FS outperforms the other methods, providing a highly accurate and reliable classification of brain tumors. The Hybrid (CNN + EfficientNetB0) model also performs well, while the Fine-Tuning Hybrid struggles significantly, as reflected in both Kappa and MCC scores.

In the second data set in Table 2, the CNN method was considered first. This method had precision values of 79%, 84%, 88% and 81% in the classification of no tumor, glioma tumor, pituitary tumor and meningioma tumor classes, respectively. The overall accuracy rate was calculated as 83.97%. The EfficientNetB0 model had precision values of 99%, 98%, 99% and 94% in the classification of no tumor, glioma tumor, pituitary tumor and meningioma tumor classes, respectively. The overall accuracy rate of this method was calculated as 96.86%. The hybrid model (CNN + EfficientNetB0) had precision values of 86%, 97%, 99% and 86% in the classification of no tumor, glioma tumor, pituitary tumor and meningioma tumor classes, respectively. The overall accuracy rate was calculated as 92.50%. The fine-tuned hybrid model (Fine Tuning Hybrid) had precision values of 93%, 96%, 96% and 98% in the classification of no tumor, glioma tumor, pituitary tumor and meningioma tumor classes, respectively. The overall accuracy rate was calculated as 95.64%. Finally, the proposed EfficientNetB0 model and the QGA-FS method had precision values of 96%, 99%, 98% and 99%, respectively, in the classification of no tumor, glioma tumor, pituitary tumor and meningioma tumor classes. The overall accuracy rate was calculated as 98.25%. In summary, the Cohen’s Kappa score and MCC both confirm that the proposed EfficientNetB0 with QGA-FS model provides superior performance with strong classification reliability and accuracy compared to other models, highlighting its potential for clinical applications. In general, the EfficientNetB0 model and QGA-FS method proposed for brain tumor classification have the highest accuracy rate compared to other methods and stand out as the most effective and reliable solution in distinguishing tumor types. This shows that the proposed method exhibits superior performance in automatic classification of brain tumors and can make a significant contribution in clinical applications.

Figure 4 shows the comparison of accuracy and loss values of different brain tumor classification models of the first data set throughout the training process. With the proposed EfficientNetB0 model, QGA-FS achieved the highest accuracy and lowest loss values, offering the best performance among all models.

Figure 5 compares the confusion matrices and receiver operating characteristic (ROC) curves of brain tumor classification models. QGA-FS with the proposed EfficientNetB0 model has the highest accuracy and discrimination ability for all classes and shows the best results in the ROC curve. This reveals that the proposed model offers the best performance and performs more accurate classification than other methods.

Figure 6 compares the accuracy and loss curves of brain tumor classification models for the second data set. With the proposed EfficientNetB0 model, QGA-FS achieved the highest level of accuracy and the lowest levels of loss values, demonstrating the best performance among all models. These results show that the proposed model is the most effective and reliable solution for brain tumor classification.

Figure 7 compares the confusion matrices and ROC curves of the brain tumor classification models in the second data set. With the proposed EfficientNetB0 model, QGA-FS has the highest accuracy rate and lowest error rate for all classes and shows near-perfect results in the ROC curve. This reveals that the proposed model offers the best performance and performs more accurate classification than other methods.

Comparison with state-of-the-art deep learning based techniques

Table 3 compares various deep learning techniques used in brain tumor classification. The accuracy rates of the QGA-FS with the proposed EfficientNetB0 model were found to be quite high compared to other methods. Hybrid deep learning approaches developed by Raza et al. (2022) aim to increase classification accuracy by using multiple methods. However, these models often fail to achieve ideal performance due to dataset limitations or architectural complexity. For example, Raza et al. achieved an accuracy rate of 96.30%. In contrast, the EfficientNetB0 with QGA-FS model, which uses QGA for feature selection, achieved higher accuracy rates of 98.36% (Figshare) and 98.25% (MRI). This demonstrates that our model achieves better performance without requiring more complex architectures, thanks to its high accuracy and efficient feature selection capability.

Tummala et al. (2022) and Aloraini et al. (2023) achieved high accuracy rates (98.70% and 99.10%, respectively) using transformer-based hybrid models. However, the main disadvantage of these models is their high computational cost and resource requirements. The proposed EfficientNetB0 model offers lower computational cost compared to transformer models while achieving competitive accuracy rates, thanks to QGA-based feature selection. This makes our model more suitable for practical use.

Studies such as Swati et al. (2019) and Pundir & Rajeev Kumar (2021) used transfer learning methods, performing classification on MRI data using transferred knowledge. However, the lack of originality and lower accuracy rates (89.90% and 91.80%) are significant drawbacks. In contrast, the proposed model combines EfficientNetB0’s powerful feature extraction capacity with QGA, offering a more innovative and effective approach compared to these transfer learning models.

Models like Shyamala & Brahmananda (2023), which use feature reduction techniques, improve performance on high-dimensional datasets but still show lower accuracy rates (94.70%). While this study used relief-based feature selection, the proposed QGA-based feature selection model provides a more effective feature selection and higher accuracy due to its broader search space. This demonstrates that our model performs better even on more complex datasets.

Compared to other methods in the table, the proposed EfficientNetB0 Model with QGA-FS stands out in terms of originality, high accuracy rates, and effective feature selection (Shaik & Cherukuri, 2022; Malla, Sahu & Alutaibi, 2023; Sahoo et al., 2023; Deepak & Ameer, 2021). By combining QGA-based feature selection with EfficientNetB0’s strong feature extraction capabilities, we achieved the highest accuracy rates in brain tumor classification. Additionally, we provide a more efficient and faster model by avoiding the high computational costs of transformer-based models. These results show the potential for our model to be used in clinical applications.