A hybrid deep learning approach with progressive cyclical CNN and firebug swarm optimization for breast cancer detection

Significant contributions

The significant contributions of this study are as follows:

• 1.

  Patch-wise segmentation is employed to divide large mammography images into smaller, localized patches, enabling finer feature extraction and improved detection of subtle abnormalities.

• 2.

  A hybrid feature extraction strategy is proposed, combining deep features from a pre-trained ResNet model with handcrafted features, including LBP (texture), Hu moments (shape), and edge maps, to enhance diagnostic accuracy.

• 3.

  A novel progressive cyclical convolutional neural network (P-CycCNN) is introduced, incorporating progressive training with cyclical learning rate scheduling to improve convergence stability and generalization.

• 4.

  FSO is used to optimize hyperparameters such as learning rate, dropout, and regularization, outperforming traditional tuning approaches.

Data collection

A publicly available dataset of digitised mammography pictures with labelled annotations for breast cancer detection, the Curated Breast Imaging Subset of the Digital Database for Screening Mammography (CBIS-DDSM), was utilised in this work. This dataset is a subset of the larger Digital Database for Screening Mammography (DDSM), comprising breast images annotated with benign and malignant lesions from various hospitals.

Data source link You can access the dataset at the following.

URL: https://www.cancerimagingarchive.net/collection/cbis-ddsm/

Database name: Curated Breast Imaging Subset DDSM dataset

ID number: calc_case_description_test_set.csv

The dataset includes two mammography images: Cine and post-processed. A label indicating whether the lesion is benign or malignant is attached to each photograph. Each lesion’s type, location, size, and other ground truth information are labelled on each image. A lesion’s classification—whether benign, malignant, or unclassified—is also included in the labelling. The DICOM format is commonly used for medical imaging and can be easily converted to other standard image formats, such as PNG or JPEG, for further processing. The CBIS-DDSM dataset is divided into training and testing subsets to guarantee generalisability during model evaluation. It covers a range of lesion types, including masses, calcifications, and asymmetries. Sample images are shown in the Fig. 3.

Preprocessing

Preprocessing is essential to ensure the quality of the images and prepare them for the feature extraction and model training stages. All images are resized to a fixed dimension (e.g., 256 × 256 pixels) to ensure uniformity of input size for the deep learning models. This resizing helps maintain a consistent aspect ratio while making the images manageable for computational processing.

(1) $$I_{resized}=f(I_{original},256,256)$$where $I_{original}$ is the original image, and $I_{resized}$ is the resized image. Pixel values are normalized to the range [0, 1] to avoid large input values hindering the training process. This normalisation ensures that the model’s weights are updated stably during training.

(2) $$I_{normalized}(x,y)=\frac{I_{resized}(x,y)}{255}$$where $I_{resized}(x,y)$ is the pixel intensity of the resized image at the position $(x,y),$ and $I_{normalized}(x,y)$ is the normalized pixel value. Local contrast enhancement is applied to improve the visibility of potential tumor regions, making abnormal areas in the mammogram more distinguishable from the background. Techniques such as histogram equalization or Contrast Limited Adaptive Histogram Equalization (CLAHE) are used:

(3) $$I_{enhanced}(x,y)=CLAHE(I_{normalized})$$where $I_{enhanced}(x,y)$ is the contrast-enhanced pixel value. Patchwise segmentation divides each mammogram into smaller patches to improve feature extraction and model efficiency. The sliding window technique is used to extract non-overlapping image patches from the resized and enhanced images:

(4) $$P_{i,j}=f(I_{enhanced},W,H)$$where $P_{i,j}$ represents the $i,j\text{-}th$ image patch of size $W\times H,$ extracted from the enhanced image $I_{enhanced}$. This allows the model to focus on localized features and detect abnormalities at a finer resolution. Figure 4 shows the preprocessing pipeline block diagram.

To enhance generalization and mitigate overfitting, standard augmentation techniques such as random rotation, zoom, flipping, and translation are applied dynamically during training.

To prevent overfitting and improve generalization, various data augmentation techniques are applied to the image patches, including Random rotations between $-{30}^{\circ }and{30}^{\circ }$, Random zooming by a factor of $0.8to1.2$, Random horizontal flipping. Random translations within 10% of the image size. These augmentations are applied on the fly during model training to create variations in the dataset:

(5) $$P_{augmented}=f_{augmentation}(P_{i,j})$$where $P_{augmented}$ is the augmented image patch obtained by applying one or more transformations to $P_{i,j}$. To ensure all features are on a similar scale and improve convergence during training, both deep features (from CNNs) and handcrafted features (such as texture or shape features) are standardised:

(6) $$f_{scaled}=\frac{f_{feature}-\mu }{\sigma }$$where $f_{feature}$ is a feature value, $\mu$ is the mean, and $\sigma$ is the standard deviation of the feature values across the dataset. The dataset is divided into training, validation, and testing sets. The labels corresponding to benign or malignant lesions are preserved throughout the preprocessing steps. A standard 70-30 split is used for training and testing. Cross-validation (e.g., 5-fold cross-validation) is used during model training to improve generalization and prevent overfitting:

(7) $$TrainingSet=\{X_{train},Y_{train}\},TestingSet=\{X_{test},Y_{test}\}$$where $X_{train}$ and $X_{test}$ represent the input image data, and $Y_{train}$ and $Y_{test}$ represent the corresponding labels (benign or malignant).

CAPLAHE applied contrast enhancement to improve the visibility of subtle abnormalities within the mammograms, which is critical for accurate diagnostics. Resizing the images ensures that the input dimensions are equal for the deep learning model. Figure 5 illustrates the results achieved from the preprocessing pipeline in refining the mammogram images for deep learning. Mammogram images are also segmented into smaller, diagnostically relevant areas known as patches. This strategy enables region-based feature learning within the patches, enhancing the model’s performance, sensitivity, and generalisation ability.

Patchwise image segmentation

Patch-wise image segmentation processes such as breast segmentation help divide large images like mammograms into smaller, more convenient patches. This way, the model can focus on patches in this image, allowing small changes that may have gone unnoticed during the full scan to be detected. The segmentation appears to alleviate the model’s need for extensive analysis, as it enables the model to focus on just the relevant sections. In this approach, each mammogram image I of size $W\times H$ is divided into smaller patches of size $P_w\times P_h,$ where $P_w$ and $P_h$ represent the width and height of each patch, respectively. The total number of patches $N_{patches}$ that can be extracted from the image is given by the equation:

(8) $$N_{patches}=\left[\frac{W}{P_w}\right]\times \left[\frac{H}{P_h}\right]$$where $\lfloor \cdot \rfloor$ denotes the floor function, ensuring that only complete patches are considered, and the remaining pixels that do not fit into a whole patch are ignored. Each patch $P_{i,j}$ is extracted from the image using a sliding window technique, with coordinates $(i,j)$ representing the position of the patch within the image grid. Mathematically, the extraction of each patch can be defined as:

(9) $$P_{i,j}=I[(i-1)\cdot P_w:i\cdot P_w,(j-1)\cdot P_h:j\cdot P_h]$$where $P_{i,j}$ represents the $i-th$ row and $j\text{-}th$ column patch. The indices $[(i-1)\cdot P_w:i\cdot P_w]$ and $[(j-1)\cdot P_h:j\cdot P_h]$ define the range of pixels along the image’s width and height for each patch.

Depending on the chosen step size, the sliding window approach can be overlapping or non-overlapping. When the step size $S_w$ for the horizontal direction and $S_h$ for the vertical direction are equal to the patch size $P_w$ and $P_h$, the patches do not overlap. For a non-overlapping sliding window, the equation remains:

(10) $$P_{i,j}=I[(i-1)\cdot P_w:i\cdot P_w,(j-1)\cdot P_h:j\cdot P_h].$$

However, if the step sizes $S_w$ and $S_h$ are smaller than the patch sizes, the patches will overlap. In this case, the equation for an overlapping sliding window is:

(11) $$P_{i,j}=I[max(0,(i-1)\cdot S_w):min(W,(i-1)\cdot S_w+P_w),max(0,(j-1)\cdot S_h):min(H,(j-1)\cdot S_h+P_h)].$$

This equation satisfies the desired conditions, as it minimizes the extracted patches moving out of the domain of the image by using the min and max functions (boundaries in the data). Then, based on the provided cancerous tissues dataset, each extracted patch is labelled according to the presence of cancerous tissues. If a patch covers a malignant zone (tumor), it will be categorised as malignant; if it does not, it will be classified as benign. The label is conditional on what most of the area comprises, and the labels for all patches are based on that particular class. Patchwise segmentation is a method in which large mammogram images are segmented into smaller parts, helping with focused attention so that each patch can be analysed in detail. This allows the model to capture the most granular local features, allowing it to excel at finding minor irregularities in breast tissue. Apart from that, patch-wise segmentation reduces overhead by processing smaller regions, making it more straightforward to train models on large datasets without sacrificing detection accuracy. The stepwise procedure for patchwise image segmentation is provided in Algorithm 1.

Hybrid deep feature extraction

The hybrid approach makes use of the strengths of deep learning approaches as well as ‘traditional’ hand-crafted features to enhance women’s chances of surviving breast cancer detected in mammogram images. For this purpose, a CNN such as residual network (ResNet), which is already trained to extract deep-meaningful features from an image is used. The deep features include the central and most important components learned from the picture, such as those with skin and fur. Other features comprise hand-crafted ones, such as those relating to texture, shape, and edge, which are known to be helpful in the context of medical image interpretation. Combining both types of features, the model incorporates the high-level features learned by the CNN and the low-level features obtained through hand-crafting methods and enhances performance in detecting malignant areas. ResNet stands for residual network and is a famous deep learning architecture that can train intense networks by relying on the connections called “residuals,” which help avoid the vanishing gradient problem. It contains several convolutional layers and residual blocks, which are more efficient for feature extraction. For breast cancer detection, a pre-trained ResNet-50 model is utilised to extract high-level deep features from images derived from mammography.

Handcrafted features such as local binary patterns (LBP) and Hu moments capture local texture and shape descriptors that are often lost in deep abstraction layers. By fusing these with deep CNN features, the proposed hybrid method benefits from both semantic representation and low-level granularity, improving lesion detection accuracy in images with variable contrast, density, and scale—common challenges in mammography.

The process of deep feature extraction using ResNet can be described mathematically as follows: Let I represent the input mammogram image of size $W\times H$, which has been preprocessed (resized, normalised, etc.). Pass/through the ResNet model:

(12) $$F_{deep}=ResNet-50(I)$$where $F_{deep}$ is the output feature map after passing through the $ResNet$ network, the features are typically extracted from intermediate layers, such as the final convolutional layer, before the fully connected layers. This allows the model to capture both low- and high-level features. The ResNet model uses residual blocks to add the input to the output of a block, which ensures better information flow through the network, leading to the effective extraction of complex features like tumor shapes, edges, and high-level patterns from the mammogram images.

Specially designed features are also constructed from the mammogram images to extract non-deep features like the images’ texture, shape, and edge features. These features have been traditionally applied in medical imaging, and when combined with deep features, they are often sufficient for additional discrimination. Image coarseness describes the qualitative aspects of structures within the imagery, in other words, aspects of the textural patterns in the image and its spatial relationships. A key texture descriptor is the LBP, which captures texture by considering pixel regions and the intensity values of surrounding pixels. The LBP operator is defined as:

(13) $$LBP=\sum _{n=0}^{P-1}s(I(x,y)-I_n)2^n$$where P is the number of neighboring pixels, $I(x,y)$ is the intensity of the central pixel, and $I_{\mathrm{n}}$ is the intensity of the neighboring pixels. The function $s(x)$ is defined as:

(14) $$s(x)=\{\begin{array}{c}1ifx\ge 0\\ 0ifx<0.\end{array}$$

LBP encodes the local spatial patterns in the image, which are often helpful in distinguishing different tissue types in mammography images. Shape-based features, such as Hu moments or Zernike moments, capture the global shape of objects (e.g., tumor boundaries) in the picture. The Hu moments are invariant to translation, rotation, and scale and are computed as follows:

(15) $${\mu }_{p,q}=\sum _{x=1}^W\sum _{y=1}^H{(x\text{-}\overline{x})}^p(y\text{-}\overline{y}{)}^{q}I((x,y))$$where $I(x,y)$ is the pixel intensity, and $\overline{x}$, $\overline{\mathrm{y}}$ are the image center coordinates. Hu moments are a set of seven invariants derived from these moments, capturing global shape information, such as the roundness of masses in mammography images. Edge-based features help detect boundaries between different tissues. One common technique for edge detection is the canny edge detector, which identifies areas with rapid intensity changes. The edge map is generated as follows:

(16) $$E(x,y)=Canny(I(x,y))$$where $E(x,y)$ is the edge map produced by the Canny operator, it highlights the regions of high-intensity change, which correspond to potential tumor boundaries in mammograms.

Progressive cyclical convolutional neural network

The progressive training strategy begins with a simplified CNN configuration (e.g., fewer layers or frozen parameters) to first learn low-level features such as edges and textures. As training progresses, additional layers are unfrozen in a staged manner, allowing the model to learn increasingly complex features (e.g., tumor boundaries or irregular shapes). This curriculum-inspired strategy improves convergence and generalisation. Complementing this, the cyclical learning rate schedule oscillates between a minimum and maximum bound across training cycles, helping escape local minima and accelerating convergence. Figure 6 shows the P-CycCNN architecture.

Let $f_{simple}$ represent the simple network at the initial stage, and $f_{complex}$ represent the more complex network added at a later stage. The network at each progressive stage $k$ can be expressed as:

(18) $$f_K=f_{simple}+f_{complex}$$where $f_{simple}$ contains essential convolutional layers, and $f_{complex}$ represents additional layers added in later stages to capture more abstract patterns. In the initial stages, only the layers of $f_{simple}$ are trained, while $f_{complex}$ is frozen. As the model progresses, more layers are unfrozen and gradually trained to capture more complicated features. Cyclical learning rates (CLR) adjust the learning rate dynamically during training. Instead of using a fixed learning rate throughout the training, the learning rate is periodically adjusted within a defined range. This technique helps the network avoid local minima and promotes the exploration of a broader parameter space, leading to better convergence and faster training. The cyclical learning rate schedule follows a sinusoidal pattern where the learning rate is increased from a lower bound. $LRmin$ to an upper bound $LRmax$, and then decreased back to the lower bound.

The cyclical learning rate can be defined mathematically as:

(19) $$LR(t)=L{R}_{min}+\frac{1}{2}(L{R}_{max}-L{R}_{min})\left(1+\cos \left(\frac{T_{max}}{T_{cur}}\pi \right)\right)$$where, $LR(t)$ is the learning rate at time step $t,L{R}_{m}in$ is the minimum learning rate, $L{R}_{max}$ is the maximum learning rate, $T_{cur}$ is the current training iteration within the cycle, $T_{max}$ is the total number of iterations within one cycle. The learning rate fluctuates between $L{R}_{min}$ and $L{R}_{max}$ over each cycle, allowing the model to escape local minima and potentially find better solutions during training. To implement a P-CycCNN, we start with a pre-trained CNN (VGG16) and use progressive training. Initially, only a subset of the layers is trained. As the model progresses, more layers are unfrozen and trained using cyclical learning rates. The cyclical learning rate adjusts the learning rate dynamically, helping the model converge faster and avoid overfitting. The CNN layers use kernels (filters) to convolve the input image and extract features. The kernel values in the initial layers are typically small, learned through the training process. These kernels are learned to capture fundamental patterns such as edges, textures, and other local features. As the network progresses, the filter values become more complex, capturing abstract patterns and object shapes.

The convolution operation with a kernel K at a given layer is defined as:

(20) $$F(x,y)=(I\ast K)(x,y)=m=\sum _{m=-W/2}^{W/2}\sum _{m=-H/2}^{H/2}I(x+m,y+n)K(m,n)$$where, $I(x,y)$ is the input image, $K(m,n)$ is the kernel at position $(m,n)$, W and H are the dimensions of the kernel, $F(x,y)$ is the output feature map at location $(x,y)$. In the case of a convolutional layer, the kernel values are learned during training, and the model adjusts the kernel weights using backpropagation to minimize the loss function.

The proposed P-CycCNN enhances traditional adaptive learning rate strategies like Adam, RMSProp, and cosine annealing in two ways. First, P-CycCNN employs cyclical learning rates that span between a minimum and maximum oscillation value. This oscillation allows the optimizer to escape local minima and saddle points much easier. On the contrary, Adam and RMSProp use monotonic decay functions which can lead to convergence stagnation. Second, the progressive scheduling approach to training within P-CycCNN starts by ‘unfreezing’ layers, which are simpler at the beginning (textures and edges) and progresses towards more complex patterns subsequently (shapes and boundaries). This approach mirrors human curriculum learning, enhancing convergence stability alongside generalization. Due to these changes, P-CycCNN is especially useful for fine-grained distinguishing tasks in medical imaging.

Firebug swarm optimization for hyperparameter optimization

This is a new metaheuristic FSO based on the movement of firebugs that gravitate towards stronger light sources, which represent better fitness solutions. The firebugs in the swarm serve as candidate hyperparameters, including learning rate, dropout rate, and batch size. The swarm evolves by allowing firebugs with poor fitness to follow those with better fitness based on the validation set. FSO conducts a more intelligent and adaptive search than random and grid search hyperparameter methods. This study focused on utilising automated methods to minimise manual effort while maintaining consistency across various model folds. Key parameters included the cyclical learning rate bounds, dropout regularisation rate, and network depth, all of which were precisely fine-tuned, leading to optimisation. Consequently, the models converged more quickly, exhibited lower variance in training losses, and improved classification metrics without imposing high computational demands. This directly enhances the effectiveness and robustness of the P-CycCNN framework developed.

The incorporation of FSO as a firebug-inspired warm light movement mechanism for hyperparameter optimisation in the deep learning model for breast cancer detection using mammography marks a novel contribution of this work. The incorporation of firebug swarm optimization (FSO) as a firebug-inspired warm-light movement mechanism for hyperparameter optimization in the deep learning model for breast cancer detection using mammography marks a novel contribution of this work and is detailed in Algorithm 2. Other works have applied metaheuristics like PSO, GA, or Firefly, but FSO is unique in that it incorporates a dynamic light-intensity movement mechanism based on a firebug’s foraging behaviour. Thus, FSO improves the trade-off between exploration (searching new areas of the solution space) and exploitation (optimising known reasonable solutions). Unlike PSO, which depends on global best particles, or the firefly algorithm, which relies on fixed brightness decay, FSO’s locally adaptive swarm behaviour yields better convergence in higher-dimensional parameter spaces. Its application to optimise parameters of cyclical learning rate, dropout, and depth of the neural network in our proposed hybrid model is an unexplored contribution in the literature.

Various swarm-based optimisers, such as PSO and the firefly algorithm, have been studied in the context of medical image analysis. However, FSO employs a distinctive approach inspired by firebug swarms as they navigate towards light signals. Unlike previous algorithms, FSO adjusts its movement strategy based on the local attraction dynamics of the swarm, thereby enhancing exploration of the hyperparameter space. This work marks the first application of FSO in the classification of breast cancer in mammograms, thus addressing the gap in intelligent model tuning.

Assessment metrics

Based on the broad evaluation criteria: numerical metrics, graphical analyses, and comparison with other models, the proposed model seems to be efficient, accurate, and dependable for diagnosing patients with breast cancer from mammograms. It also has good precision, recall, and F1-score; therefore, it has the potential for clinical application to reduce the extent of misdiagnosis and enhance patients’ outcomes.

To assess various performance metrics for the efficiency of the detection model for breast cancer, the following were considered:

Accuracy: The correct percentage of mammogram images that have been classified correctly.

Loss: The deviation from the predicted values during training and subsequent testing.

Precision: The proportion of the true positive predictions to all positive predictions made with an emphasis on minimising false positives.

Recall the proportion of the cases which were true positives to those that were actual positives out of all cases. Emphasis is placed on minimizing the false negatives.

F1-score: Weighted average of precision and recall measured geometrically while emphasising the recall aspect of the score.

AUC-ROC: The performance of the model in differentiating between classes of attributes. This suggests that the more the auc, the better the performance.

Training, validation, and testing outcomes

The performance outcomes were explained in terms of visualisations and numerical summaries, enabling easy understanding. Important observations include:

Efficacy measures:

The training set accuracy was at 99% confidence level.

The test set accuracy was 98

Both datasets’ good prediction and recall statistics imply strong classification power with no evidence of overfitting.

Training time and resource utilization:

An NVIDIA GTX 1080 Ti GPU was utilised to train for fifty epochs with a batch size of thirty-two.

Model comparison:

The proposed model performed better than other improved versions of CNNs and even some that had optimizations such as data augmentation and hyperparameter tuning across all purposes.

Visual analysis of performance

Accuracy and loss curves:

The accuracy curve expanded during the epochs, indicating that the model was learning successfully.

A reduction of observed error rates was set in without the problem of overfitting, and this is verified with a decreasing trend highlighted in the loss curve.

Precision, recall, and F1-score:

The model correctly identified cases of malignancy without classifying benign cases into malignant and positive cases. The high statistics across all measures prove this point.

AUC-ROC curve:

The model performed well, owing to its AUC of 0.95, which indicates its discrimination ability for malignity and benignity.

Confusion matrix:

The 94.5% and 96.5% classification accuracy of benign and malignant cases respectively reveals that only a few errors occurred during the classification results.

Calibration curve:

A high degree of accuracy in terms of predictions and actual events that occurred is inferred from the strong calibration achieved by the model.

Cross-validation and statistical reporting

All experiments were repeated using 5-fold cross-validation. Each model was trained and evaluated across the folds, and the final performance was reported as the mean $\pm$ standard deviation. To account for training instability, three independent runs were conducted per fold, and 95% confidence intervals were computed using bootstrapping.

The performance of the model metrics achieved with 5-fold cross-validation repeated thrice, along with all other relevant statistics, is detailed in Table 6. The generalization performance is quite good with average accuracy of 97.8% ( $\pm$0.6), precision 94.9% ( $\pm$0.5), recall 97.2% ( $\pm$0.4), and F1-score at 96.0% ( $\pm$0.5). In addition, the model’s ability to differentiate between malignant and benign cases is robust, as evidenced by the AUC of 0.952 ( $\pm$0.007). All statistical measures yielded narrow confidence intervals which denotes the high stability and reliability of the P-CycCNN + FSO architecture with different splits of data.

Ablation study

Table 7 presents an ablation study evaluating the incremental effect of each proposed enhancement. Adding handcrafted features improved the base CNN’s performance by enriching local texture and shape information. Introducing progressive training led to better feature generalization by gradually unfreezing layers. Finally, the cyclical learning rate schedule further stabilised convergence and boosted overall accuracy. These results confirm that each module makes a significant contribution to the model’s overall performance.

Comparison with related works

The existing approaches, compared to the proposed one, for classifying breast cancer are presented in Table 8. While the support vector machine (SVM) model done by Shravya, Pravalika & Subhani (2019) achieved an accuracy of 92.7%. Sivapriya et al. (2019) reported the highest accuracy of 99.76% using a random forest classifier. Such performance, however, may be dataset-specific or prone to overfitting. The use of deep learning models also reflects standard accuracies observable in histopathological analyses. For instance, Mahmud, Mamun & Abdelgawad (2023) used ResNet50 with an accuracy of 90.2%, which confirms this notion.

The accuracy metrics of the proposed hybrid deep learning model which incorporates P-CycCNN and FSO shows 98% accuracy achieving high accuracy with strong generalization at the same time. Unlike the traditional models, this model combines shallow and deep features and uses modern techniques for optimization and training, thus making it more resilient across multiple mammography datasets. This proves its high practicality and highlights why it outperforms compared methods in the literature.

Limitations of the study

The study has several limitations. The performance of the P-CycCNN model relies on the dataset’s size and quality, and its ability to extend to new general problems or clinical settings remains unclear. Its training was done only on mammography images; thus, its usage is restricted to this kind of imaging. Due to high complexity and computational requirements, the model may be unsuitable for implementation in low-resource settings, and overfitting can also be a concern. Unclear models of AI could equally negatively impact clinician confidence, while the requirement for frequent revisits and retraining makes scaling much more challenging. Furthermore, the evaluation criteria of the model should be interpreted carefully since they may not depict the clinical outcome, and,other issues such as integration into existing health organisations, data privacy, loss of confidentiality and legal approval may also be a barrier. Ultimately, biases introduced by the dataset may potentially impact the model’s fairness; thus, improvements must be made for even wider applications.