Deep feature fusion for melanoma detection using ResNet-VGG networks with attention mechanism

Pre-trained CNN models

CNNs are deep feed-forward architectures known for their remarkable performance in feature extraction and classification tasks (Saleem et al., 2024). Their architecture integrates convolutional, pooling and fully connected layers, enabling efficient hierarchical feature extraction. In this study, two pre-trained models, ResNet50 and VGG16, are used for feature extraction, selected for their robust architectures and proven accuracy in large datasets such as ImageNet.

Proposed framework

This study uses a unified framework with feature extraction, attention mechanism and classification. By combining pre-trained models with innovative attention modules, the proposed framework solves the main challenges in binary classification tasks: the lack of distinguishable features and redundant data. Deep features were extracted using two pre-trained models, ResNet50 and VGG16, which use different architectural paradigms to capture different feature representations of the input images. A feature fusion approach was used to reduce the redundant and meaningless component that results from the fusion of features in multiple models. It is simply a way to combine the strengths of ResNet50 and VGG16 while preserving the important features. To focus the model on critical data, we added attention mechanisms to refine the fused features. The workflow of the proposed model is shown in Fig. 2.

After the feature maps are independently extracted from ResNet50 and VGG16, they are flattened and concatenated into a single feature vector that retains both global and local representations. This fused feature is then successively refined with the channel and spatial attention modules. The channel attention module focuses on which features are important by adaptively reweighting the channels based on the global average and maximum pooled descriptors and boosting critical texture and lesion-specific filters while suppressing irrelevant ones. Next, the spatial attention module determines “where” to focus by analysing the mean and maximum projections of the feature maps along the channel axis, helping the model to highlight lesion-relevant spatial regions. These refined features are then passed to the final fully linked layers for binary classification. This integration allows CA-SLNet to utilise complementary feature hierarchies from both backbones while dynamically refining attention to discriminative content, improving sensitivity to subtle lesion patterns in different inputs.

Feature fusion strategy

The feature fusion strategy combines the feature maps extracted by ResNet50 and VGG16 by flattening their outputs into 1D vectors. These vectors are then concatenated along the channel dimension to form a unified feature vector that includes global and local image details. The channel and spatial attention modules are applied sequentially to enhance the fused features. Channel-based attention highlights significant channels, while spatial attention highlights prominent spatial regions in the combined feature map. This ensures a rich, focused feature representation for subsequent processing.

To validate the discriminative power of the fused feature embeddings, we performed a t-SNE visualization of the high-dimensional feature space generated by the proposed model. Figure 3, the feature embeddings form different clusters corresponding to the Ph², ISIC-2016 and ISIC-2017 datasets. The clear separation between these clusters demonstrates the model’s ability to create compact and distinguishable feature representations, thus improving classification performance.

Dataset

The ISIC-2016 dataset is an important resource in dermatology and computer vision. It is often used to further develop and evaluate machine learning algorithms for skin cancer detection based on dermoscopic images. It consists of dermoscopic images categorized as melanoma and benign cases. In addition, based on the ISIC archive, which contains images with different resolutions, a diverse set of skin lesion images was used for the study. The ISIC-2016 dataset contains a total of 1,279 images. The ISIC-2017 dataset includes 2,000 images of skin lesion cases. This dataset serves as an important benchmark for the development and validation of machine-learning models for skin cancer diagnosis. Due to the high prevalence of keratoses, the images are often divided into two main categories: benign and malignant. The Ph² dataset was also included in this study (Benyahia, Meftah & Lézoray, 2022). It contains 200 dermoscopic RGB images. This dataset, curated at the Hospital Pedro Hispano in Matosinhos, contains both dermatoscopic images and clinical assessments by dermatologists indicating whether an image represents a normal case, a melanoma, or an atypical nevus. The ISIC-2016, 2017 and Ph² datasets evaluate melanoma detection as they offer a large variety and with good annotations. Dermatologists have manually annotated all images in these datasets with precise basic annotations. To assess class balance, we analysed the distribution of benign and malignant cases in all datasets. The ISIC-2016 dataset consists of 900 benign and 379 malignant samples, while ISIC-2017 contains 1,400 benign and 600 malignant cases. The Ph2 dataset contains 160 benign and 40 malignant images. These ratios reflect inherent class imbalances, especially in ISIC-2016 and Ph², which may bias model learning towards the majority class. Therefore, we implemented customised augmentation and attention mechanisms to improve sensitivity and reduce misclassification, especially false negatives in minority classes.

Design and optimization of CA-SLNet for skin lesion classification

In this section, we develop and optimize the proposed CA-SLNet for skin lesion classification. To improve the generalization, data augmentation techniques such as rotation, flipping and color jittering were applied to the data. The ResNet50 and VGG16 models are combined with attention modules for refined feature extraction. Binary cross-entropy loss, an Adam optimizer and a StepLR scheduler for learning rate decay were used in training. They quickly improve the model’s ability to be classified and to classify. The detailed flowchart in Fig. 4 shows the stages of the proposed framework, from image acquisition to the final classification process.

Training environment

All training and evaluation experiments were performed on a dedicated local workstation equipped with an NVIDIA RTX 4070 GPU, an Intel Core i9 processor and 32 GB RAM running Windows 11. The model was implemented with Python 3.10 and PyTorch 2.1.0. Image preprocessing and augmentation were performed with Torchvision 0.16.0, while performance metrics and analysis were performed with scikit-learn 1.3.0 and Matplotlib 3.7.2. GPU acceleration was managed with CUDA 11.8. This environment ensured consistent model training, uniform evaluation and reproducibility of results.

Training hyperparameters

The CA-SLNet model was trained using the Adam optimiser with a learning rate of 0.001, a batch size of 32 and a maximum of 100 epochs. All input images were resized to 224 $\times$ 224 $\times$ 3. A dropout layer with a rate of 0.5 was applied before the last fully concatenated layers to prevent overfitting. Weight initialization was used when initialising the weights and early stopping was applied with a patience of 10 epochs based on validation loss to prevent overtraining. No planner was used for the learning rate. These hyperparameters were kept consistent across all data sets to ensure a fair and reproducible evaluation, as shown in Table 1.

A five-fold cross-validation was performed for each experiment to ensure the robustness of the results and to avoid bias in the assessment. Each data set was randomly divided into five equal folds, with four folds used for training and validation 80% and one fold retained for testing 20%. An internal split of 80:20 was made within the training area to form the training and validation sets. The splitting process was stratified by class to maintain the original distribution of benign and malignant samples across all folds. The performance metrics were averaged across all five folds to obtain stable and generalised results.

Proposed model algorithm

The proposed algorithm CA-SLNet (channel attention for skin lesion network) describes a systematic approach to melanoma detection that utilises attention mechanisms for robust feature extraction and classification Algorithm 1. By incorporating preprocessing, feature fusion and feature refinement into the attention method, the algorithm provides high accuracy, strong capabilities and feature generalisation in melanoma classification. The operation of CA-SLNet consists of several important steps: First, the data is pre-processed by normalising the input images to ImageNet mean and standard deviations. Data augmentation techniques such as random flipping, rotation and affine transformations are applied to the data to artificially augment and improve the generalisation of the model.

In the following phase, feature extraction is performed using two deep learning architectures, ResNet50 and VGG16. The feature maps generated by these backbones are linked to form a comprehensive feature representation. After feature extraction, attention mechanisms are used to refine the concatenated features. Attention to the channels is calculated by combining the global average pooling and global max pooling operations, followed by fully concatenated layers to assign weights to the important channels. At the same time, spatial attention is computed by applying a convolution operation to the combined average and max feature maps to highlight critical spatial regions for melanoma detection. These attention mechanisms work in tandem to produce a refined feature representation.

In the classification phase, the features are flattened and fed into a fully concatenated layer with a sigmoid activation function that generates the final prediction probabilities. Training is performed by iteratively optimizing the model parameters according to the BCE loss objective function. Second, efficient convergence of the weights is achieved by using the first and second-order moments of the gradient updates provided by the Adam optimizer. A learning rate scheduler (StepLR) keeps the learning rate stable and optimal throughout the training. The algorithm systematically employs feature fusion, attention mechanisms and robust optimization to ensure that the final model has accurate classification and discrimination capability. CA-SLNet integrates spatial and channel attention and effectively highlights critical lesion regions to improve its robustness in different datasets and clinical scenarios.

Evaluation metrics

The evaluation metrics used were accuracy, precision, recall, F1-score and the area under the receiver operating characteristic (ROC) curve. The metrics for precision and recall are defined as follows:

(9) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}},\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}},$$where TP, FP and FN are the number of true-positive, false-positive and false-negative results.

The F1-score, a harmonic mean of precision and recall, is computed as:

(10) $$\mathrm{F}1\text{-}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\cdot \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\cdot \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}.$$

The area under the curve-receiver operation characteristic (AUC-ROC) curve measures the discriminatory ability of the model:

(11) $$\mathrm{A}\mathrm{U}\mathrm{C}={\int }_0^1\mathrm{T}\mathrm{P}\mathrm{R}(\mathrm{x})\cdot \mathrm{d}\mathrm{x},$$

The true-positive rate depends on the false-positive rate $x$, which is noted as $\mathrm{T}\mathrm{P}\mathrm{R}(x)$. This model guarantees its performance and reliability in detecting melanoma. It has been tested with the ISIC 2016, 2017 and Ph² datasets.

To assess the performance of the model, we used different evaluation metrics, such as confusion matrices, to assess accuracy along with the receiver operating characteristic (AUC). The model used cross-validation to obtain a reliable and generalized assessment of performance. The efficiency of the model was improved by carefully selecting several hyperparameters, in particular, the batch size, the loss function, the learning rate, the target size and the optimization function.

Ablation study

The ablation study evaluates how the different parts and parameters of the CA-SLNet architecture change the model’s capacity. The primary focus of this analysis is to identify the essential operations in the design of signal processing, such as attention mechanisms, feature fusion and hyperparameters, to achieve optimal performance. In this way, attention mechanisms have increased the importance of focusing on appropriate components to conceal interference upstream. By turning off the channel attention (CA) module, the performance degradation in ISIC-2016, ISIC-2017 and Ph² was 3.25%, 2.98% and 3.85%, respectively. Switching off the spatial attention (SA) module led to a reduction of up to 4.2%. These results show that attention mechanisms are crucial for improving feature representation and increasing discriminative capacity. Therefore, using ResNet50 and VGG16 features was very important for both overall and local features. The average accuracy of ResNet50 and VGG16 is 5.6% and 6.4%, respectively, showing that feature fusion is crucial for proper visualization.

Hyperparameter tuning was crucial for optimal performance. A learning rate of $1\times {10}^{-5}$ achieved the best balance between stability and convergence, as higher rates ( $1\times {10}^{-4}$) led to instability and lower rates ( $1\times {10}^{-6}$) slowed down training. A batch size of 32 balanced computational efficiency and accuracy, while a dropout rate of 0.5 effectively curbed overfitting. As shown in Table 2, these configurations were experimentally validated as the most effective for training CA-SLNet. Using pre-trained ResNet50 and VGG16 models was critical for achieving high performance. Replacing the pre-trained weights with random initialization decreased accuracy by more than 10% for all datasets, with corresponding reductions in AUC values. This underlines the importance of transfer learning for improving generalization and accelerating convergence. These results confirm the robustness and optimality of the CA-SLNet architecture for automatic melanoma detection.

Feature localization and explainability in CA-SLNet

To assess the interpretability of the CA-SLNet model, we created visual explanations using Grad-CAM overlays, saliency maps and attention heat maps. These visualisations help to understand how the model prioritises different image regions when making classification decisions. Figure 5 illustrates this process using a skin lesion as an example. The base feature maps of ResNet50 and VGG16, Figs. 5B and 5C, show a general feature extraction, while the fused CA-SLNet feature map, Fig. 5D, localises the lesion-critical regions more precisely. The channel attention map, Fig. 5E, highlights the feature dimensions that have the greatest influence on classification, while the spatial attention map, Fig. 5F, identifies the lesion-relevant regions in the spatial domain. The saliency map Fig. 5G illustrates pixel-level influence and the Grad-CAM overlay Fig. 5H provides an intuitive heatmap that confirms the model’s focus on lesion boundaries and morphological features. These interpretation tools increase the transparency of CA-SLNet and show that its predictions are based on medically meaningful image regions, which is crucial for clinical validation.

Comparison with state-of-the-art methods

The proposed CA-SLNet performs better than existing state-of-the-art (SOTA) models across multiple dermatological benchmark datasets, including ISIC-2016, ISIC-2017 and Ph². For the classification, the results of the ISIC-2016 dataset showed the robust performance of the model. Figure 6 shows a high accuracy and correct classification of 900 benign and 361 malignant samples, with no false positives and only 18 false negatives. The minimal misclassification confirms that the model will reliably identify melanomas in clinical applications. The effectiveness of the model is shown in Fig. 7. The model showed excellent performance in classifying cancerous lesions with an average AUC value of 0.9814 $\pm$ 0.0121. The individual AUC values obtained in the five cross-validation folds ranged from 0.9643 to 0.9968, demonstrating the reliable and consistent performance of the model. These results suggest that the model could be a robust model for dermatologists to aid clinical diagnosis. Table 3 compares the classification accuracy of CA-SLNet with previous work on the ISIC-2016 dataset. The table summarizes the performance of the models developed in recent years and their respective accuracy levels.

Misclassification analysis

To analyse the model errors, we examined the confusion matrices for each dataset. In ISIC-2016, the model misclassified only 18 malignant cases as benign, with zero false positives. In ISIC-2017, 64 malignant cases were classified as benign, reflecting the visual complexity of the dataset and the overlap of lesions. In contrast, the model in the Ph2 dataset achieved near-perfect classification, with only one benign image incorrectly categorised as malignant. These results indicate that false-negative images were the main cause of misclassification, particularly in ISIC-2017. Our augmentation strategy and attention-based fusion mitigated the imbalance between classes and improved overall recall, but further improvements may be needed to reduce critical misclassifications.

Model inference time and hardware efficiency

To assess the clinical suitability of CA-SLNet, we measured the inference time and hardware efficiency with an NVIDIA RTX 4070 GPU and an Intel Core i9 processor. The model processes a dermatoscopic image in 42 ms, enabling near real-time support. Despite its dual backbone design (ResNet50 and VGG16), efficiency is maintained through selective layer tuning. CA-SLNet has a model size of 97 MB, 29.6 million parameters and uses 1.8 GB of GPU memory during inference, confirming its suitability for use in clinical environments with moderate hardware.

The proposed model was developed to accurately discriminate between melanoma and non-melanoma skin lesions.

Table 3 shows that CA-SLNet achieves an accuracy of 98.59% and outperforms most other models, including the original work in Amin et al. (2020) with 98.39%. CA-SLNet exhibits high accuracy and provides several additional advantages by incorporating the ResNet50 backbone and the VGG16 backbone and attention mechanisms in feature extraction. In particular, it is generalized through an advanced data augmentation technique. These features make the classification robust and adaptable to different datasets, making this tool an actionable tool for melanoma classification. The CA-SLNet achieves the highest possible accuracy in discriminating melanoma and its accuracy is commercially attractive and state-of-the-art.

Figure 8, the model shows a high classification accuracy for the ISIC-2017 dataset with correct identification of 1,400 benign and 536 malignant samples. The model also achieved 0 false positives and 64 false negatives, which is important as it shows the robust predictive ability. This reflects that the model avoided overdiagnosis, which is crucial for avoiding unnecessary treatment. While the slightly increased rate of false-negative results provides an improvement to minimize the number of missed melanoma cases, this is important in clinical applications. The mean AUC value from Fig. 9 is 0.9626 $\pm$ 0.0215. The AUC value demonstrates that CA-SLNet shows strong performance in discriminating benign from malignant skin lesions, reinforcing its position as a reliable model for skin cancer diagnosis. The individual AUC metrics of five cross-validation splits ranged from 0.9317 to 0.9873, indicating a stable and reliable performance of the model. The model classifies the images with high accuracy and an evident capacity to distinguish between skin diseases and other conditions. This underlines the great potential of the model to help dermatologists in clinical practice. Table 4 shows the performance of the latter studies as well as their gradual advancement in melanoma classification sensitivity.

In addition, the results shown in Figs. 8 and 9 demonstrate the perfect predictive power of the model with a high AUC and minimal false positives. The CA used for dermoscopy performed better than others because its attention planes extracted more features. However, they only focus on important parts of the dermoscopic images. In addition, the model created in this work is more general due to the application of new methods for data augmentation, e.g., random deletion and color shifting. This is reflected in the consistently high performance of CA-SLNet over a given dataset, which in turn emphasizes the robustness of the network for use in clinical practice.

Figure 10 shows that the model correctly identified 40 malignant cases and correctly classified 159 benign lesions, with only one misdiagnosis of benign as malignant and no misdiagnosed malignant cases. The model achieves its results through significant accuracy with almost no errors in classification. The model achieves exceptional performance by completely suppressing misdiagnosis of melanoma, as the elimination of false negatives contributes significantly to early detection of melanoma and improved patient outcomes.

The evaluation of the Ph² dataset of the proposed model shows both reliable and valid results. The model showed an average AUC value of 0.9814 $\pm$ 0.0121 in Fig. 11, indicating its significant ability to discriminate between benign and malignant lesions. The AUC values measured in five-fold cross-validation ranged from 0.9644 to 0.9969, confirming the reliability of the model.

For all three datasets examined, ISIC-2016, ISIC-2017 and Ph², the model achieved a very similar AUC, which could be due to specific characteristics of the concrete datasets, including the degree of disparity between classes and the range of lesions, as well as the overall resolution of the images. The results of these datasets showed that the proposed model is fully generalizable and has a high classification accuracy and a nearly flawless AUC metric, which is excellent for clinical use. Several challenges were encountered when working with the different datasets. The model can provide accurate and reproducible results when overcoming these challenges. It supports the accuracy of using a dermatoscopy in different clinical cases related to melanoma. Table 5 shows the classification accuracy of CA-SLNet compared to other models for the Ph² dataset. Current models for melanoma classification have shown promising results, as seen in the table and the CA-SLNet proposed in this work has the highest accuracy among the models.

The proposed CA-SLNet model shows exceptional performance and achieves 99.50% accuracy on the Ph² dataset compared to SOTA, as shown in Table 6. This performance sets a new benchmark, exceeding the highest previously reported accuracy of 99.10% (Benyahia, Meftah & Lézoray, 2022). This significant improvement is due to CA-SLNet’s attention mechanisms, which increase focus on key lesion regions and a robust data augmentation strategy that improves generalisation. Figure 12 highlights the exceptional performance of the proposed model across multiple datasets. CA-SLNet consistently outperforms previous approaches, achieving near-perfect classification results in the Ph² dataset and strong generalization in the ISIC 2016 and ISIC 2017 datasets. These results confirm the robustness and adaptability of the model and emphasize its potential for practical applications in clinical dermatology.

The integration of attentional mechanisms allows the model to extract more informative features from the lesion regions, ensuring higher precision, recall and F1-scores. Consequently, CA-SLNet proves to be a reliable solution for accurate and fast melanoma detection, which is promising for real-world applications.

Figure 12 illustrates the numerical results shown in Table 6 and shows that the model is able to achieve state-of-the-art results on different data sets. This makes CA-SLNet a scalable and suitable solution for real-world clinical applications where high precision and generalization are required.

Statistical reliability of results

To validate the robustness and reproducibility of the CA-SLNet model, we performed a five-fold cross-validation for each dataset. In all datasets, ISIC-2016, ISIC-2017 and Ph², the model consistently provided high performance with low variance. The mean AUC values were reported along with their standard deviations in the respective ROC curve analyses: 0.9814 $\pm$ 0.0121 for ISIC-2016, 0.9626 $\pm$ 0.0215 for ISIC-2017 and 0.9814 $\pm$ 0.0121 for Ph². These small standard deviations reflect minimal variation in classification ability across different folds and demonstrate the statistical significance and reliability of the model’s predictions.