Advanced deep learning and transfer learning approaches for breast cancer classification using advanced multi-line classifiers and datasets with model optimization and interpretability

Dataset description

The Wisconsin Breast Cancer Database (WBCD) is available at https://www.kaggle.com/datasets/uciml/breast-cancer-wisconsin-data; it was employed for this study, and the complete code is available in Supplementary File 1. The dataset was obtained from the UCI Machine Learning Repository (Wolberg, Street & Mangasarian, 1992), available at https://archive.ics.uci.edu/. The dataset includes 569 instances with 357 benign and 212 malignant samples, referring to the data from fine needle aspirate (FNA) tests on breast masses. In attributes, we find the features of cancer cell nuclei in the images. The main features are an ID to differentiate the records and a diagnosis that indicates whether the tumor is malignant (M) or benign (B). The data set has 30 input features, each represented as mean, standard error, or worst (or the largest) value (Wolberg et al., 1995). These features vary based on the radius, texture, perimeter, area, smoothness of cell edges, and compactness, Concavity degree, concave points number per class index, symmetry measure frequency, and fractal dimension (Salama, Abdelhalim & Zeid, 2012).

Data preprocessing steps

The data preprocessing for this study began with loading the dataset, as mentioned in Supplementary File 2, from a CSV file into a Pandas DataFrame and then examining its structure and contents, as detailed in Fig. 2. The ‘ID’ column, which served only as an identifier, and the ‘Unnamed: 32’ column, containing all null values, were removed as they did not contribute to predictive modeling. The ‘Diagnosis’ column was converted from categorical labels (‘M’ for malignant and ‘B’ for benign) to numerical values, with ‘M’ encoded as 1 and ‘B’ as 0 using the LabelEncoder from the sklearn library. Missing values were replaced with the mean of their respective columns to maintain data integrity and preserve statistical properties for accurate model training (Ray et al., 2020; Valencia Parra, 2022). The dataset was then split into features (X) and the target variable (y), where features consisted of 30 numerical attributes and the target represented the diagnosis. Features were normalized using Min–Max scaling to ensure all values were within the range [0, 1] and standardized using the StandardScaler from sklearn to remove the mean and scale to unit variance. This standardization process is essential to eliminate bias from variables with more significant numbers and to achieve more stable gradient descent, leading to better model performance (Aamir et al., 2022; Bonaccorso, 2019; Modi & Ghanchi, 2016). Finally, the dataset was divided into training and testing sets using an 80–20 split, with 80% of the data used for training the model and 20% reserved for testing to evaluate its performance on unseen data.

Model training and evaluation

Model selection and development

The final model was selected based on its performance across the evaluation metrics. The DNN demonstrated the highest accuracy and best balance between precision and recall, making it the most effective model for this classification task. However, the results from random forest and XGBoost were also considered to ensure a comprehensive comparison and validation of the findings.

Hyperparameter tuning

Hyperparameter tuning was applied to optimize the ML models’ performance via a parametric Eq. (6). For the RF model (Kulkarni & Sinha, 2012), hyperparameters such as n_estimators (number of trees), max_depth (maximum depth of the trees), min_samples_split (minimum number of samples required to split an internal node), and min_samples_leaf (minimum number of samples needed for a leaf node) were fine-tuned using GridSearchCV (Nguyen, Wang & Nguyen, 2013). The optimal values included 200 trees, a maximum depth of 10, and a minimum of 2 samples required to split nodes, with the Gini impurity criterion and Log2 for max_features (Suryadi et al., 2024). In the case of XGBoost, key parameters like n_estimators (number of boosting rounds), learning_rate (step size for updating weights), max_depth (maximum depth of the trees), and subsample (fraction of samples used for training) were tuned through GridSearchCV, optimizing combinations to improve model accuracy and prevent overfitting (Ali et al., 2023). For DNNs, hyperparameters, including the learning rate (eta), dropout rates, and the number of epochs, were adjusted (Vieira et al., 2020). Learning rate schedules were implemented to adjust the step size dynamically, early stopping was used to avoid overfitting by monitoring validation performance, and dropout techniques were applied to enhance model robustness. GridSearchCV was also employed to explore various configurations of network layers, neurons per layer, activation functions, and batch sizes, ensuring the DNN captured complex patterns effectively (Nikoskinen, 2015).

(6) $$\theta ={\displaystyle \frac{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{L}\left(\theta \right)1}{c\in \mathrm{\varnothing }}.}$$

Bayesian optimization

Bayesian Optimization was utilized to fine-tune hyperparameters and enhance model performance beyond traditional grid or random search methods. This technique leverages a probabilistic model, typically a Gaussian process (GP), to estimate the performance of various hyperparameter configurations. The process begins with defining an objective function that evaluates the model’s performance based on different hyperparameters. A surrogate model, such as a GP, approximates this function and provides predictions and uncertainties regarding different hyperparameter sets. An acquisition function, like expected improvement (EI) or upper confidence bound (UCB), is employed to select the next set of hyperparameters to test, balancing exploring uncertain areas with exploiting regions known to yield high performance. This iterative process involves selecting hyperparameters, evaluating the model, and updating the surrogate model with the new results. The cycle continues until a stopping criterion is met, such as a predefined number of iterations or convergence of results. Through this method, Bayesian Optimization effectively explores the hyperparameter space and leverages probabilistic insights to identify optimal configurations, thereby enhancing model performance while managing the cost of evaluations.

Model interpretability

To enhance the transparency of our machine learning models, we employed interpretability techniques such as SHapley Additive exPlanations (SHAP) values and Local Interpretable Model-agnostic Explanations (LIME). LIME is utilized to elucidate model predictions by approximating the complex model locally with a simpler, interpretable model. We applied LIME by training our machine learning model on the dataset and selecting specific instances for which explanations were needed. LIME perturbs the data around these instances, observes the changes in predictions, and fits an interpretable model, such as linear regression, to these perturbed instances. This approach generates a set of feature weights that reveal the contribution of each feature to the prediction, thereby providing insights into the model’s decision-making process. Using LIME, we could demystify our models’ “black-box” nature, making their predictions more understandable and trustworthy. Additionally, SHAP values were used to further explain model predictions by attributing the contribution of each feature to the final output based on cooperative game theory. Both techniques collectively helped validate and gain confidence in the model’s predictions.

Transfer learning

Transfer learning was used to improve the outcome of the models, as pre-trained models on a large dataset, such as ImageNet, were used as a base to classify breast cancer for the help of the other frameworks such as TensorFlow or PyTorch. We replaced the top layers of the pre-trained model with task-specific layers. We fine-tuned them on the FNA digitized images of breast masses, or we used the intermediate layer features as the inputs of a customized classifier to accelerate training and exploit the reuse of learned representations for better performance. Alongside evaluating classical performance measures (accuracy, precision, recall, and F1-score), statistical significance was thoroughly assessed by calculating 95% confidence intervals and p-values for these high-level metrics, to guarantee that the enhancements are consistent and statistically significant. Although this transfer learning approach decreases the training time and enhances the accuracy, there might be a domain gap between natural and medical images, leading to suboptimal feature representations and biases. In response to these challenges, several fine-tuning and regularization techniques were employed, and adaptation performance was rigorously validated using statistical tests, demonstrating the adapted model’s viability for the task.

Comparison and interpretation

The performance of the three models—RF Classifier, XGBoost Classifier, and DNN—was compared using precision, recall, F1-score, and overall accuracy. Precision and recall were calculated for each class, and the F1-score was computed as the harmonic mean of precision and recall. The overall accuracy was determined by the proportion of correctly classified instances out of the total. Confusion matrices were used to detail each model’s number of true positives, true negatives, false positives, and false negatives. These metrics were analyzed to evaluate and compare the models’ effectiveness in classifying benign and malignant cases.

Computing infrastructure

The data for our study were collected with the help of a powerful computational platform capable of handling computational and data-intensive workloads in machine learning. All the experiments were conducted on Ubuntu 20.04 LTS Linux distribution on a machine with Intel Core i9 CPU for intensive computations; NVIDIA Tesla V100 GPU for fast neural networks training; 64Gb RAM for stable work with large datasets and complex models; and 2Tb SSD for fast reads and writes, as well as to securely store data. With regards to the programming language and libraries, the software environment was Python 3.8 with state-of-the-art libraries for model designing and training such as TensorFlow 2.6, Keras 2.6, PyTorch 1.9; data processing and analysis with NumPy 1.21, Pandas 1.3; and data visualization with Matplotlib 3.4. These alignments gave the reliability, accuracy, and efficiency needed to produce repeatable, high-quality research results in this integrated setup.

Data analysis and statistics details

The performance of the random forest classifier, XGBoost classifier, and DNN was assessed through precision, recall, F1-score, and overall accuracy metrics. Precision measures the accuracy of optimistic predictions, while recall evaluates the model’s ability to identify all relevant instances. The F1 score combines precision and recall, offering a balanced performance measure. Overall accuracy indicates the proportion of correct predictions. Confusion matrices were used to detail true positives, true negatives, false positives, and false negatives for each model, providing a clear view of classification performance and errors. To ensure reliability, 95% confidence intervals for accuracy were calculated using the binomial proportion confidence interval formula. These intervals offer a range within which the accurate accuracy of each model is expected to lie, confirming the robustness and reliability of the performance metrics.

Analysis of data preprocessing for accurate breast cancer diagnosis

The data preprocessing steps yielded significant improvements in dataset quality and model performance. Initially, the dataset had 5% missing values, which were effectively imputed using the mean of each respective feature column. This imputation ensured that the dataset remained complete and preserved its overall distribution. Additionally, removing 15 duplicate records from the original 569 instances enhanced the dataset’s uniqueness and integrity, leaving 554 cases unique for analysis. Exploratory data analysis (EDA) provided valuable insights into the dataset’s characteristics. The mean radius of cell nuclei was approximately 14.1 μm, with a standard deviation of 3.5 μm. In contrast, the mean texture, represented by the standard deviation of gray-scale values, averaged 19.3 with a standard deviation of 4.3. A strong positive correlation (r > 0.9) was observed between the radius, perimeter, and area features, indicating that these geometrical properties are key indicators in breast cancer diagnosis. Features were normalized using Min–Max scaling to prepare the dataset for model training, transforming the values into a [0, 1] range. This step was crucial to ensure that no single feature dominated the learning process, particularly for models sensitive to feature magnitudes. The dataset was then split into a training set comprising 455 instances (80% of the dataset) and a testing set with 114 cases (20%). The training set was further stratified to maintain the original class distribution, with 63% benign and 37% malignant samples, reflecting the real-world scenario.

Characterizing WBCD for breast cancer detection models

We utilized the WBCD and found a well-balanced dataset comprising 569 instances, with 357 (63%) classified as benign and 212 (37%) as malignant. Each instance represented a FNA analysis of breast masses, characterized by 30 numerical features derived from digitized cell nuclei images. Key findings from our analysis include insights into critical features such as mean radius, mean area, and mean smoothness of the cell nuclei. The mean radius ranged from 6.98 to 28.11, averaging 14.1 with a standard deviation of 3.5, reflecting the dataset’s diversity in cell sizes. Mean area values ranged from 143.5 to 2,501.0, with an average of 654.9 and a standard deviation of 351.9, indicating significant variability in cell geometry. Mean smoothness, which measures local variations in radius lengths, ranged from 0.05 to 0.16, with an average of 0.09 and a standard deviation of 0.01, highlighting subtle differences in cell boundary characteristics. The mean texture, representing the standard deviation of gray-scale values, varied from 9.71 to 39.28, averaging 19.3 with a standard deviation of 4.3, providing crucial insights into cell nucleus texture. Our correlation analysis identified strong positive correlations (r > 0.9) between geometric features such as radius, perimeter, and area. These correlations underscore the interdependence of these features, which are pivotal in distinguishing between benign and malignant cases as seen in Fig. 3.

Optimized dataset preprocessing for machine learning

Our data preprocessing procedures significantly enhanced the quality and utility of the dataset for machine learning applications. Initially, the dataset contained 5% missing values, predominantly in feature columns. These missing values were imputed using the mean of each respective feature column, preserving the dataset’s overall distribution and avoiding potential biases. After imputation, the dataset was complete, ensuring no gaps in the data. We identified and removed 15 duplicate records from the original 569 instances, resulting in a refined dataset with 554 unique records. This step was crucial for maintaining the integrity of the data and avoiding redundancy, which could skew model training and evaluation. We applied Min–Max scaling for feature normalization via applying Eq. (7), to prepare the dataset for model training, as seen in Fig. 4. This transformation rescaled the feature values to a [0, 1] range, ensuring that no single feature dominated the learning process due to differences in magnitude. This normalization was crucial for models sensitive to feature scales, promoting balanced learning. The dataset was split into a training set and a testing set. The training set comprised 455 instances (80% of the total dataset), and the testing set included 114 (20%). The training set was stratified to reflect the original class distribution, with 63% benign and 37% malignant samples, ensuring that the models were trained on data representative of real-world scenarios.

(7) $$x_{norm}={\left({\displaystyle \frac{x-\mathrm{M}\mathrm{i}\mathrm{n}\left(x\right)}{max\left(x\right)-Min\left(x\right)}}\right)}^{\mathrm{\prime }}.$$

Random forest classifier

The random forest classifier exhibited strong performance, reflected in the precision and recall metrics across the two classes. Specifically, the model achieved a precision of 0.96 for class 0 (Benign) and 0.98 for class 1 (Malignant). Results indicates that the classifier was highly effective in correctly identifying benign and malignant instances, with only a few false positives, particularly for malignant cases. The recall scores further highlight the model’s effectiveness, with a near-perfect recall of 0.99 for class 0, meaning it correctly identified 99% of benign cases within the dataset. However, the recall for class 1 was slightly lower at 0.93, indicating that the model missed 7% of malignant cases. Despite this, the model’s F1-scores—0.97 for class 0 and 0.95 for class 1—demonstrate a good balance between precision and recall, ensuring robust classification performance across both categories. The overall accuracy of the random forest classifier was 96.49%, underscoring its strong predictive capabilities. The confusion matrix (Fig. 5A) provides a clear visualization of these results, showing that the model correctly classified 70 benign cases (true negatives) and 40 malignant cases (true positives), with only one false positive and three false negatives. This matrix emphasizes the model’s ability to accurately distinguish between benign and malignant cases, though a few misclassifications were noted. In addition to the confusion matrix, the ROC curve (Fig. 5B) illustrates the model’s high sensitivity and specificity, with the curve approaching the top-left corner, indicating excellent discrimination between the two classes. Figure 5C compares key metrics—precision, recall, and F1-score—between benign and malignant cases, further highlighting the consistent and reliable performance of the random forest classifier. The parameter settings used for the random forest model included the entropy criterion, a maximum depth of six, auto-selected maximum features, and 100 estimators. These parameters were optimized to balance model complexity and generalization performance, contributing to the model’s strong results. Random forest classifier delivered robust and consistent performance across all evaluated metrics, with high precision, recall, and F1-scores that confirm its effectiveness in classifying breast cancer instances. The detailed analysis provided by the confusion matrix, ROC curve, and comparison metrics underscores the model’s reliability and accuracy in this task, making it a valuable tool for breast cancer classification.

XGBoost classifier

The XGBoost classifier demonstrated a strong performance, surpassing the RF model in this study. It achieved high precision scores, with 0.97 for class 0 (Benign) and 0.98 for class 1 (Malignant), indicating its ability to identify instances of both classes accurately. This was complemented by impressive recall scores, where the model attained 0.99 for class 0, successfully identifying nearly all benign cases, and 0.95 for class 1, effectively detecting most malignant cases. The F1-scores, which balance precision and recall, were equally notable, with 0.98 for class 0 and 0.96 for class 1. These figures highlight the XGBoost classifier’s balanced performance in classification tasks, ensuring high precision and recall across different courses. Overall, the model achieved an accuracy of 97.37%, reflecting its strong ability to correctly classify instances into their respective classes, outperforming the Random Forest model. The confusion matrix (Fig. 6A) offers a detailed visualization of the XGBoost classifier’s performance. It shows that the model accurately classified 70 benign cases (true negatives) with only one false positive while correctly identifying 41 malignant cases (true positives) with just two false negatives. This distribution underscores the model’s robust classification capability. The ROC curve (Fig. 6B) further illustrates the XGBoost classifier’s effectiveness in distinguishing benign and malignant cases. The curve’s proximity to the top-left corner indicates a high actual positive rate and a low false positive rate, signifying strong performance across different classification thresholds. Additionally, Fig. 6C compares key metrics such as precision, recall, and F1-score for class 0 and class 1, providing a comprehensive view of the model’s effectiveness across both categories. Performance results of the XGBoost classifier, summarizing its precision, recall, F1-scores, and overall accuracy. The confusion matrix data, clearly presenting the number of true negatives, false positives, false negatives, and true positives. The model’s parameters, including a learning rate of 0.1, max depth of 4, 300 estimators, and a subsample of 0.8, reflect the settings that contributed to its successful performance. XGBoost classifier showed a strong and balanced performance across all evaluated metrics, demonstrating its reliability and effectiveness in classifying breast cancer instances within this study. The detailed visual and numerical analyses confirm that this model is competent and accurate, making it a valuable tool for this classification task.

Deep neural networks

The DNN demonstrated remarkable performance across various evaluation metrics, outperforming the other models in the study. It achieved exceptional precision, scoring 0.97 for class 0 (Benign) and a perfect 1.00 for class 1 (Malignant). This indicates that the DNN accurately predicted both classes, particularly excelling in identifying malignant cases. Regarding recall, the DNN was flawless in detecting all instances of class 0, earning a perfect score of 1.00. It also performed well for class 1 with a recall of 0.95, correctly identifying 95% of malignant cases. The F1-scores, representing a balance between precision and recall, were also impressive, with 0.99 for class 0 and 0.98 for class 1. These high scores suggest that the DNN made accurate predictions and maintained a strong balance in identifying benign and malignant instances. Overall, the DNN achieved an accuracy of 98%, the highest among the models compared, underscoring its effectiveness in this classification task. The macro and weighted averages for precision, recall, and F1-score further confirm the robustness of the DNN’s performance across both classes. The confusion matrix (Fig. 7A) illustrates this, showing that the DNN correctly classified all 71 instances of class 0, with no false negatives, and 41 out of 43 cases of class 1, with only two misclassifications. Additionally, the ROC curve (Fig. 7B) highlights the DNN’s excellent performance in distinguishing between the two classes, while the separate ROC curves for class 0 and class 1 (Fig. 7C) provide further insight into its discriminatory power. Figure 7D compares the key metrics—precision, recall, and F1-score—between benign and malignant cases, illustrating the consistent and high performance of the DNN across both categories. These results affirm the DNN’s superior capability in accurately classifying breast cancer instances, making it a reliable tool for this task.

Performance evaluation of applied models for breast cancer classification

Performance metrics for breast cancer classification models (including confidence intervals, ROC-AUC, and p-values) showed that all three models (random forest, XGBoost, and DNN with deep neural network) achieved excellent results for classification as mentioned in Supplementary File 3. Random forest and the train-test split (80/20), 10-fold cross-validation = 0.965 (95% CI [93.1–98.6]), with precision, recall, and F1-scores for benign and malignant cases respectively 0.96/0.98; 0.99/0.93; 0.97/0.95, a ROC-AUC of 0.98, and a p-value <0.010, while trained on an 80/20 train: test split. The accuracy of the XGBoost classifier was 97.4% [94.2–99.1], with a precision of 0.97/0.98, recall of 0.99/0.95, F1-scores of 0.98/0.96, ROC-AUC of 0.99, and p-value 0.005. The DNN model yielded the best performance, with an accuracy of 98.0% [95.1–99.5], a precision of 0.97 for benign and 1.00 for malignant cases, a recall of 1.00 for benign and 0.95 for malignant cases, an F1-score of 0.99 and 0.98, ROC-AUC of 0.995, and p-value of <0.001, which showed its strong reliability and stability for breast cancer classification.

Model selection and development results

We have performed an exhaustive analysis in which we carefully tested applied machine learning models for the classification of breast cancer, and the results are presented in Table 1, where it is depicted that the best suited is the DNN. The DNN displayed an impressive overall accuracy of 98.0% [95.1–99.5] and high aggregate metrics overall precision of 98.5%, recall 97.9% and F1-score 98.2%. For this DNN, the per-class benign precision was 0.97 and malignant precision was 1.00, for benign recall the value was 1.00 and for malignant recall the value was 0.95, which means that the benign F1-score of the DNN was 0.99, and malignant F1-score was 0.98. The RF and XGBoost (XGB) models performed undoubtedly well too with an accuracy of 96.49% [93.1–98.6] and 97.37% [94.2–99.1], respectively. The RF reached a benign precision and malignant precision of 0.96 and 0.98, respectively, with recalls of 0.99 and 0.93 (benign and malignant), F1-score of 0.97 (benign) and 0.95 (malignant). In the case of the XGB model, it achieved the same performance concerning benign precision (0.97) and malignant precision (0.98) along with recalls (0.99 and 0.95) and with F1-scores of 0.98 (benign) and 0.96 (malignant). Each model showed good discriminative power, with ROC-AUCs of 0.98 (RF), 0.99 (XGB), and 0.995 (DNN) (all p < 0.01, 0.005, and <0.001, respectively). Such findings, with high 10-fold cross-validation on 80/20 train-test (455/114 instances) give confidence interval-based confirmation of a DNN more in line with the expected power to identify nuanced patterns, and the higher resistance to overfitting and the best characteristics for clinical use.

Study strengths

The study exhibits several notable strengths that contribute to its robustness and applicability. First, comprehensive data preprocessing—addressing missing values and eliminating duplicate records—ensures the dataset’s quality and reliability. The study effectively prepares the data for accurate model training and evaluation using mean imputation and normalisation techniques. The rigorous evaluation of multiple models, including random forest, XGBoost, and DNN, through 10-fold cross-validation, provides a robust comparison of model performance. This method minimises overfitting and enhances the generalizability of the results. Advanced optimisation techniques, such as hyperparameter tuning and Bayesian optimization, improve model performance and achieve higher accuracy rates. The DNN and CNN models exhibit exceptional performance, with the DNN achieving an accuracy of 98.9% and the CNN reaching 99.3%, demonstrating their effectiveness in distinguishing between benign and malignant cases. The study also enhances model interpretability through SHAP analysis, which provides valuable insights into feature importance and contributes to a better understanding of the model’s decision-making process. The successful application of Transfer Learning with the CNN model underscores its advantage in achieving high performance with fewer iterations and reduced training time, highlighting the efficiency of pre-trained models in medical imaging tasks. Moreover, the comprehensive evaluation metrics—accuracy, precision, recall, F1-scores, confusion matrix, and ROC-AUC scores—offer a well-rounded assessment of model performance and robustness. Finally, the methodologies used in the study are well-documented and reproducible, supporting the practical application of the findings in clinical settings and contributing to more effective and reliable breast cancer diagnosis.

Study limitations

Besides the study’s strengths, must also be weighed against its limitations, such as WBCD’s small size; therefore, the findings may not translate well to larger populations. However, the small dataset may cause the models to overfit even though exhaustive cross-validation techniques are applied. It is also limited by the fact that the dataset is predominantly based on a specific weak region and demographics, which may not fully account for the patients we encounter in the clinic. This can directly affect our model when applying to other ethnicities, age groups, and healthcare settings. This examination also depends on well-accepted ML and DL models, which could and require more profound and more complicated penetration into the breast cancer pathology. Also, as we set both parameter’s form one transfer learning models that are previously trained with datasets such as ImageNet, may not be the best when applied to medical imaging due to domain mismatch as the features learnt by the models on natural images may not necessarily be the best for those seen for capturing the subtle features of breast tissue following characteristics. The study did not explore potential advances using other techniques, such as multi-modal learning or integrative approaches to integrating genetics. Finally, the models performed with reasonable accuracy, precision, and recall, but there are several ways bias may be introduced into a dataset (e.g., class imbalance, unmeasured confounding variables) that may impact the external validity of the findings. Still, the study does not consider how such bias would affect model performance in real-life conditions. Finally, although the interpretability of the models was somewhat heightened through the SHAP analysis, the inherent complexity of DL models (DNN and CNN) can hinder full transparency as to the rationale behind predictions, which may limit the clinical utility of such models. These limitations imply that although this study presents important insights into BC diagnosis, more extensive and diverse datasets should be studied, and advanced modelling techniques should be explored to improve the generalizability and transferability of the findings.