Improving robustness in X-ray image classification through attention mechanisms in convolutional neural networks

Self-supervised learning (In-domain pre-training)

In the first stage of our framework, we implement a self-supervised learning (SSL) strategy to pre-train the model using unlabelled X-ray images. Instead of using weights from models trained on natural RGB datasets, such as ImageNet, we adapt the network by selectively unfreezing only the final convolutional blocks. This approach enables the model to learn domain-specific features, such as bone density, anatomical structures, and grayscale contrast characteristics, which are unique to radiographs. There are two widely adopted strategies in SSL: freezing selected layers of the pre-trained model and end-to-end fine-tuning. In our framework, we adopt the layer-freezing approach. To retain useful low-level representations, we freeze the first 50% of the convolutional layers, which include early and mid-level filters responsible for capturing edges and textures. The remaining layers are fine-tuned to focus on high-level features relevant to musculoskeletal abnormalities, such as bone fractures. To facilitate domain adaptation, we apply a transform-based pretext task using targeted image augmentations. These include horizontal flipping, small rotations ( $\pm$10°), contrast normalisation, and random cropping, each designed to introduce variability without compromising anatomical structures. During training, different augmented views of the same image are assigned the same label, encouraging the model to learn invariant and domain-specific features. The network is optimised using categorical cross-entropy loss. This SSL process produces robust pretrained weights, which are then used to initialise the full model for subsequent supervised training focused on fracture and abnormality classification.

Self-attention mechanism

The attention mechanism allows DL models to prioritise the most informative regions within an image by modelling the relationships between spatial features. This is particularly beneficial in medical imaging tasks, such as fracture detection using the MURA dataset and chest X-ray classification, where accurate diagnosis often depends on subtle contextual cues. By comparing anatomical structures across different parts of an image, attention mechanisms help the model to detect fine-grained abnormalities that might otherwise be overlooked.

At the core of this mechanism is the concept of self-attention, which enables the model to assess and weigh the relevance of each input element relative to the others. This process involves the following steps:

• For each input position $x_i$ in a sequence $X=(x_1,x_2,\dots ,x_n)$, the model derives three vectors:

  • –

    Query ( $q_i$), Key ( $k_i$), Value ( $v_i$)

  These are computed via trainable linear projections: (1) $$q_i=W^Q{x}_i,k_i=W^K{x}_i,v_i=W^V{x}_i$$where $x_i\in {\mathbb{R}}^{d_x}$ is the input vector at position $i$, and $W^Q\in {\mathbb{R}}^{d_q\times d_x}$, $W^K\in {\mathbb{R}}^{d_k\times d_x}$, and $W^V\in {\mathbb{R}}^{d_v\times d_x}$ are learnable weight matrices used to project the input into the query, key, and value representations, respectively. These projections enable the model to transform the input into subspaces tailored for attention computation.

• The similarity between the $i$-th query and all keys is measured using the scaled dot-product: (2) $$A_{ij}=\frac{q_i^{\mathrm{\top }}{k}_j}{\sqrt{d_k}}$$where $A_{ij}$ denotes the attention score and $d_k$ is the dimension of the key vectors used for scaling.

• The attention scores are normalised using the softmax function: (3) $$S_{ij}=\frac{\exp (A_{ij})}{{\sum }_{k=1}^n\exp (A_{ik})}$$yielding a weight matrix S that reflects the relative importance of other positions to the $i$-th element.

• Each output vector $y_i$ is computed as a weighted sum of the value vectors: (4) $$y_i=\sum _{j=1}^n{S}_{ij}{v}_j$$providing a context-aware representation that incorporates information from across the input sequence.

In our proposed framework, we enhance this mechanism using three parallel $1\times 1$ convolutional layers denoted as $\mathrm{c}\mathrm{o}\mathrm{n}{\mathrm{v}}_Q$, $\mathrm{c}\mathrm{o}\mathrm{n}{\mathrm{v}}_K$, and $\mathrm{c}\mathrm{o}\mathrm{n}{\mathrm{v}}_V$—to extract the query, key, and value feature maps from the input (Vaswani et al., 2017). The attention map is calculated by applying a softmax operation to the product of the query and key feature maps. These attention scores are then used to weight the value features in a component referred to as softmax-mult. The result is added back to the query features to preserve the original structure, forming an updated representation:

(5) $$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(\mathit{Q}\mathit{,}\mathit{K}\mathit{,}\mathit{V}\mathit{)}\mathit{=}\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{\mathit{Q}{\mathit{K}}^{\mathrm{\top }}}{\sqrt{{\mathit{d}}_{\mathit{k}}}}\right)\mathit{V}\mathit{.}$$

To ensure dimensional consistency between the input and the output, we apply a $1\times 1$ convolutional adjustment. The resulting attention-enhanced features are merged with the original input via residual addition, enabling the model to retain low-level details while benefiting from global contextual awareness.

This attention block is integrated into both the InceptionResNetV2 and Xception architectures used in our study, as shown in Fig. 1. By positioning the self-attention module between the convolutional and batch normalisation layers, we enable the model to emphasise diagnostically significant regions more effectively. This architectural refinement improves the network’s ability to model long-range dependencies, thereby enhancing its interpretability and performance on complex medical imaging tasks.

Feature fusion and feature selection

Feature fusion is designed to combine multiple complementary representations to enhance model performance, particularly in medical imaging tasks where multiscale and modality-specific information must be integrated. By incorporating attention mechanisms, the fusion process is optimised to prioritise clinically relevant regions while maintaining global contextual awareness. Complementing this, feature selection focuses on retaining the most informative components from high-dimensional data, improving computational efficiency and reducing overfitting, which ultimately leads to a more generalisable model.

In the proposed framework (Fig. 1), two convolutional neural networks, Xception and InceptionResNetV2, are independently applied to each anatomical region, resulting in four parallel pipelines: two for the humerus task and two for the wrist task. Each model is augmented with a self-attention mechanism to enhance the capture of both local and contextual features relevant to diagnostic interpretation.

Following feature extraction, the Scale-Invariant Feature Transform (SIFT) is employed to detect distinctive local patterns that remain stable under changes in scale, rotation, and illumination. For each task, deep features obtained from the two attention-based CNNs are horizontally concatenated, producing two task-specific fused representations: Fusion 1 (T1) for the humerus and Fusion 2 (T2) for the wrist. This fusion strategy preserves diverse spatial characteristics while aligning the dimensional structure of the original samples.

The fused representation integrates the high-level semantic information derived from both CNN architectures (3,584 feature dimensions) with the fine-grained local descriptors obtained via SIFT (128 dimensions), forming a comprehensive 3,712-dimensional feature vector. This combination allows the model to retain global contextual features while enriching them with structural details, such as bone edges, trabecular textures, and minor cortical discontinuity features, which are often attenuated in deep convolutional layers. In this study, incorporating SIFT proved particularly beneficial in capturing subtle musculoskeletal variations and improving cross-anatomical generalisation, where deep models alone showed limited sensitivity to fine-scale patterns. To improve computational efficiency and eliminate redundancy, Principal Component Analysis (PCA) is applied, retaining 95% of the total variance and reducing the feature space to 51 dimensions for subsequent classification.

The compact and distinctive features are visualised using t-distributed Stochastic Neighbour Embedding (t-SNE) to evaluate clustering behaviour and class separability qualitatively. The t-SNE projections confirm that the SIFT-enhanced feature space offers more precise class boundaries and improved intra-class cohesion, demonstrating its effectiveness in enhancing discriminative capacity.

The refined feature sets from T1 and T2 are combined vertically to form a unified high-dimensional feature space that captures the differences across tasks. This multitask fusion strategy enhances the model’s flexibility and supports generalisation across various anatomical regions. The consolidated feature set is then used to train several classical machine-learning classifiers, including KNN, SVM, and decision tree models. A comparative evaluation of these classifiers identifies the most effective architecture for each task. Additionally, the ensemble approach leverages its complementary strengths to achieve stable, reliable performance.

This two-stage fusion and feature-selection framework provides a comprehensive, compact, and interpretable representation that balances global semantic understanding with local structural precision, enhancing robustness and clinical relevance for musculoskeletal classification tasks.

Ensemble prediction via majority voting

The final classification output is generated using a majority voting ensemble method that combines predictions from three different machine learning classifiers: decision tree, KNN, and SVM. As illustrated in Fig. 1, it is essential to note that t-SNE and Gradient-weighted Class Activation Mapping (Grad-CAM) are used solely for post-hoc visualisation and model interpretability; they do not influence the feature selection process and are not used as inputs for the classifiers.

Before classification, the integrated feature representations go through a systematic process of feature extraction and selection. Initially, SIFT is utilised to extract robust local features and descriptors from the fused feature space. This step significantly enhances the model’s ability to identify distinguishing patterns, such as bone fractures and relevant regions, across varying scales, orientations, and imaging conditions. After feature extraction, the extracted features are normalised using min-max scaling and evaluated for dimensional consistency across all samples, ensuring they are compatible with the classifiers used in the later stages.

Each of the three classifiers is trained independently on a cleaned and structured feature set. The final predictions are determined through a majority voting strategy, which identifies the class that receives the most votes as the definitive prediction. This ensemble method reduces the impact of individual classifier biases, strengthens robustness, and improves the model’s generalisation capabilities.

In summary, this study presents a robust and scalable framework for medical image analysis that integrates self-supervised learning, attention mechanisms, SIFT-based feature selection, and ensemble decision-making. By incorporating domain-specific pre-training and context-aware feature extraction, the proposed methodology is designed to improve interpretability and support robust diagnostic decision-making. Its modular design allows adaptation to a variety of medical imaging tasks, providing a scalable foundation for future AI-driven diagnostic systems.

Data collection

This study utilised two primary datasets: the MURA dataset and a curated collection of chest X-ray images. The MURA dataset, introduced by Rajpurkar et al. (2017), comprises musculoskeletal radiographs across seven anatomical regions: elbow, finger, forearm, hand, humerus, shoulder, and wrist. Each image is labelled as either normal (negative) or abnormal (positive), resulting in a total of 40,561 images divided into separate training and test sets. Additionally, a chest X-ray dataset was utilised, comprising images from four diagnostic categories: COVID-19, normal, pneumonia, and tuberculosis. The dataset was divided into two groups: Group 1 (COVID-19 and normal) and Group 2 (pneumonia and tuberculosis). Moreover, to simulate diverse imaging conditions and anatomical variations, we employ data augmentation techniques, including rotation, scaling, translation, and contrast variation, during the training process. This approach significantly improves the model’s ability to generalise across imperfect and heterogeneous clinical data.

Experimental scenario

To evaluate the proposed framework systematically, two distinct case studies are defined based on target datasets: musculoskeletal X-rays (humerus and wrist) and chest X-rays. Correspondingly, two independent self-supervised learning (SSL) sources were constructed to support robust domain-specific representation learning for each target.

Target datasets

• 1.

  Case study 1-Bone X-ray Classification (Target and SSL source)

  This experiment focuses on binary classification (positive or negative) for two specific anatomical regions (humerus and wrist) from the MURA dataset. These tasks were selected to evaluate the framework’s effectiveness in detecting abnormalities in localised musculoskeletal areas.

  • SSL source: To facilitate domain-specific representation learning, approximately 40,000 unlabelled bone X-rays were collected from the MURA dataset, excluding humerus and wrist images. These included scans of the shoulder, elbow, hand, finger, and forearm. Additionally, 30,000 external bone-related X-rays (e.g., hip, knee, ankle) were incorporated to construct a diverse and comprehensive SSL source.

• 2.

  Case study 2-Chest X-ray Classification (Target and SSL source) This study assesses the generalisability of the proposed framework by applying it to chest X-ray images. The model was tasked with classifying scans into two distinct categories: COVID-19 versus normal, and pneumonia versus tuberculosis.

  • SSL source: A separate pretraining corpus of 50,000 unlabelled chest X-ray images was constructed from publicly available datasets, allowing the model to learn thoracic imaging features before fine-tuning.

All experiments used the Adam optimiser with an initial learning rate of 0.001 and a mini-batch size of 15, conducted over a maximum of 100 epochs. The training data was shuffled at the beginning of each epoch. All experiments were performed using MATLAB 2024a (The MathWorks, Natick, MA, USA) on a system equipped with an Intel Core i7 processor, 32 GB of RAM, a 1 TB SSD, and an NVIDIA RTX A3000 GPU with 12 GB of memory.

The average training duration for each deep learning model was approximately 148 min over 15 epochs for the Xception model, while it took about 296 min over 15 epochs for the InceptionResNetV2.

Experimental evaluation

This section outlines the experimental design and evaluation of the proposed framework for classifying musculoskeletal abnormalities, with a specific focus on tasks derived from the MURA dataset, which includes X-ray images of the humerus, wrist, and chest. In addition to describing the model configurations, we emphasise the vital role that dataset characteristics play in assessing the framework’s generalisability. We report comprehensive performance metrics, including accuracy, precision, recall, F1-score, specificity, and Cohen’s Kappa, to provide a thorough assessment of the model’s effectiveness.

Evaluation metrics

To evaluate the effectiveness of the proposed framework, a confusion matrix (CM) was employed to summarise the classification outcomes. In the context of binary classification, the confusion matrix consists of four key components:

• 1.

  True positives (TP): Correctly identified instances of the target class.

• 2.

  True negatives (TN): Correctly identified instances of the non-target class.

• 3.

  False positives (FP): Non-target instances incorrectly classified as the target class.

• 4.

  False negatives (FN): Target instances incorrectly classified as non-target.

These components serve as the foundation for computing various performance metrics. The following equations define the key evaluation measures used in this study:

(6) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}(\mathrm{A}\mathrm{c}\mathrm{c}.)=\frac{TP+TN}{TP+TN+FP+FN}.$$ (7) $$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}(\mathrm{R}\mathrm{e}\mathrm{c}.)=\frac{TP}{TP+FN}.$$ (8) $$\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}(\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}.)=\frac{TN}{FP+TN}.$$ (9) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}(\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}.)=\frac{TP}{TP+FP}.$$ (10) $$\mathrm{F}1\text{-}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}(\mathrm{F}1.)=2\times \frac{Presicion\ast Recall}{Presicion+Recall}.$$

Cohen’s Kappa equation

(11) $$\mathrm{K}\mathrm{o}=\frac{TP+TN}{TP+TN+FP+FN}.$$ (12) $$\mathrm{K}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}=\frac{(TP+FP)(TP+FN)}{{(TP+TN+FP+FN)}^2}.$$ (13) $$\mathrm{K}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}=\frac{(FN+TN)(FP+TN)}{{(TP+TN+FP+FN)}^2}.$$

(14) $$\mathrm{K}\mathrm{e}=Kpositive+Knegative.$$

Cohen’s Kappa score(Kappa.) =

(15) $$\mathrm{K}=\frac{Ko-Ke}{1-Ke}.$$

Results of the proposed framework solution

The predictive performance of each classifier is shown along with the overall results obtained from the ensemble approach. The architecture of the base model includes transformer blocks that employ a self-attention mechanism. This allows the model to dynamically focus on different regions of the input feature space, enhancing its ability to capture complex relationships and dependencies within the data.

Feature fusion was conducted by combining the outputs of two DL models for each task: Fusion 1 (T1) for the humerus and Fusion 2 (T2) for the wrist. Afterwards, a feature selection technique was applied to each fusion output to retain the most informative attributes. The refined features from both tasks were then integrated into a single, unified fusion pool. This consolidated feature set served as the input for training multiple machine learning classifiers, which were evaluated based on their ability to classify the combined humerus and wrist datasets accurately. For further analysis, we selected three classifiers: KNN, decision tree, and SVM. The classification results for both the humerus and wrist tasks are illustrated in Table 3. In the majority voting ensemble approach, each classifier casts a vote for a predicted class label, and the label receiving the most votes is selected as the final output. This strategy enhances overall accuracy by leveraging the complementary strengths of individual models while mitigating the effect of their respective weaknesses.

By combining the predictions of the Tree, KNN, and SVM classifiers through majority voting, the ensemble achieved an overall classification accuracy of 95.48% across the combined humerus and wrist datasets. This result is summarised in Table 4.

Ablation studies

This section examines three fundamental components of the proposed framework, exploring their significance and interconnections. Each aspect will be examined to highlight its unique contributions and how they collectively support the framework’s overarching objectives.

Visualisation and validation

Medical image analysis is a domain where diagnostic decisions carry critical implications. Traditionally, such decisions have relied on the expertise of human specialists, whose reasoning processes are transparent and interpretable. In contrast, CNNs, due to their deep and complex architectures, often lack inherent interpretability, making it challenging to understand the rationale behind their predictions. To evaluate the robustness of our attention-enhanced CNN model and to gain insight into its decision-making processes, we employed several explainable artificial intelligence (XAI) techniques, gradient-weighted class activation mapping (Grad-CAM), occlusion sensitivity, and t-distributed stochastic neighbour embedding (t-SNE). These methods offer complementary perspectives. Grad-CAM and occlusion sensitivity highlight spatial regions that influence predictions, while t-SNE visualises the structure of the learned feature space.

Grad-CAM works by using the gradients of output classification scores to generate heat maps that highlight the areas most significant to the model’s predictions. Regions with high gradient activations are deemed crucial to the network’s decision-making process. On the other hand, occlusion sensitivity involves systematically masking portions of the input image and observing the changes in the output. This technique helps identify which areas are essential for accurate predictions. Figure 2 provides both visual and quantitative comparisons of these two methods in humerus and wrist classification tasks, confirming that the models consistently focus on anatomically and diagnostically relevant regions.

We used t-SNE to visualise the high-dimensional fused features generated after applying attention mechanisms, feature fusion, and SIFT-based selection. By projecting these features into two dimensions, t-SNE allows us to qualitatively assess the compactness within classes and the separability between classes. It is crucial to understand that t-SNE is used solely for visualisation and does not impact on model training, feature selection, or classification decisions.

Figure 3 presents a comprehensive series of t-SNE plots that depict how the fused features evolve with different processing stages. The progression across these subfigures highlights the effectiveness of combining attention, fusion, and feature selection in generating representations that are not only robust for classification but also interpretable in terms of latent structure. The integration of these XAI techniques underscores that the model’s predictions are informed by valid, domain-relevant features, both in terms of spatial saliency and latent feature structure. This enhances confidence in the reliability and clinical applicability of the proposed approach.

Comparison with the state-of-the-art

Table 9 presents the performance of the proposed method, which demonstrates superior efficacy compared with existing approaches by integrating attention mechanisms, optimised feature selection, and a majority voting ensemble strategy. The framework has achieved state-of-the-art results, reflecting a significant improvement in classification accuracy relative to previous studies. For example, Feng et al. (2021) reported an accuracy of 70.23%, followed by subsequent research indicating continuous advancements. Rath, Reddy & Singh (2022) attained an accuracy of 80.06%, while Zou & Arshad (2024) reached 86.20%. Additional improvements were documented by Alammar et al. (2023) at 87.85% and Alammar et al. (2024), who achieved 92.71%. More recently, Ounasser et al. (2023) and Sanzida Ferdous Ruhi, Nahar & Ferdous Ashrafi (2024) reported accuracies of 90.00% and 90.48%, respectively, highlighting a consistent trend towards increasingly robust and effective frameworks for X-ray image classification.

Although the proposed framework has demonstrated state-of-the-art performance, it is important to critically evaluate its limitations and potential generalisability in real-world clinical environments. While the model has shown robust results on the MURA dataset, its performance may be influenced by external datasets or clinical archives, which may present variations in factors such as image quality, resolution, and acquisition protocols. The dataset used in this study has a relatively balanced distribution of normal and abnormal cases. However, it is essential to recognise that real-world clinical data often exhibits significant class imbalances, especially for rare conditions such as uncommon fractures or degenerative changes, which are typically under-represented in these datasets. This discrepancy between the training data and actual prevalence may result in reduced model sensitivity for low-frequency diagnostic categories, ultimately limiting the framework’s overall effectiveness and reliability in routine clinical practice.

Chest X-ray dataset

The chest X-ray dataset used in this study was publicly available. To evaluate the performance and generalisability of the proposed classification framework, the dataset was organised into two distinct groups: COVID-19, normal, pneumonia, and tuberculosis. Data augmentation techniques were applied to increase dataset diversity and mitigate the risk of overfitting. For a thorough and impartial evaluation, each group was further divided into three subsets: training, validation, and test sets. The training set was used to optimise model parameters, while the validation set facilitated hyperparameter tuning and early stopping to prevent overfitting. The test set was reserved for the final performance evaluation. This partitioning strategy ensured a comprehensive assessment of the model’s ability to generalise across various respiratory conditions, while also maintaining robustness and mitigating the risk of catastrophic forgetting.

The two groups were defined as follows:

• Group 1: COVID-19 and normal classes. This collection includes chest X-ray images classified into two categories: patients diagnosed with COVID-19 and individuals exhibiting no apparent lung pathology (normal). The dataset provides the model with an opportunity to identify and learn distinct features that differentiate COVID-19 abnormalities from healthy lung patterns. This supports binary classification within the context of viral infections.

• Group 2: pneumonia and tuberculosis classes. This collection comprises chest X-ray images indicative of pneumonia, characterised by lung inflammation and fluid accumulation, as well as tuberculosis, which frequently manifests with nodules, infiltrates, or cavitary lesions. This combination facilitates an assessment of the model’s capability to differentiate between bacterial and mycobacterial pulmonary infections.

The diagnostic categories outlined in Table 10 were employed to assess the proposed framework’s capability to generalise across a variety of respiratory conditions. This evaluation focused on the model’s ability to accurately learn, retain, and transfer knowledge across multiple classification tasks. Additionally, the assessment sought to validate the model’s robustness and its resilience to catastrophic forgetting when confronted with diverse diagnostic categories.

Results of the proposed framework solution with chest X-ray

This section outlines the experimental design and results of the proposed framework for chest X-ray classification. The dataset was divided into two groups, each comprising two diagnostic categories. The primary objective of these experiments was to assess the effectiveness of attention-based DL models, feature fusion strategies, and ensemble classification methods in accurately identifying respiratory conditions across both groups.

Initially, the Xception and InceptionResNetV2 models were trained on chest X-ray images using SSL. Both models were augmented with transformer blocks incorporating self-attention mechanisms, enabling them to focus on spatially and contextually significant regions within the images. This design enhanced the models’ ability to capture clinically relevant features.

Subsequently, feature fusion was applied to combine the learned representations from the two classes within each diagnostic group, resulting in Fusion 1 for Group 1 and Fusion 2 for Group 2. A feature selection technique was then employed to reduce dimensionality while preserving the most discriminative features from each fusion output. The selected features from both groups were merged into a unified combined fusion pool, representing a comprehensive set of patterns across the chest X-ray dataset.

This consolidated feature pool was used to train and evaluate multiple machine learning classifiers, including KNN, decision tree, and SVM. The classification performance of these classifiers is summarised in Table 11.

To improve robustness and generalisation, a majority voting strategy was implemented across the three classifiers, KNN, decision tree, and SVM. In this ensemble approach, each classifier independently predicts a class label, and the final prediction is determined by the class receiving the majority of votes. This method effectively mitigates the limitations of individual classifiers while capitalising on their complementary strengths.

The final ensemble decision, derived through majority voting, achieved an overall classification accuracy of 99.80% across the combined chest X-ray groups. This outstanding performance is summarised in Table 12. These results highlight the effectiveness of the proposed framework in reliably distinguishing between diagnostic categories, supported by a structured and interpretable learning pipeline.

Experimental evaluation of attention layer in the CNN models for ablation study 1

This section provides an overview of integrating self-attention layers into two CNN models. In this study, we applied SSL on two distinct sets of chest X-ray images. The Test set results are presented in Table 13, the Training set results are in Table 14, and the Validation set results are shown in Table 15. Upon reviewing these three tables, it is evident that the models consistently exhibit strong performance across the training, validation, and test datasets. This observation suggests effective generalisation and indicates that there has been no significant overfitting.

Experimental evaluation of deep feature fusion for ablation study 2 (Chest X-ray)

This ablation study investigated the effect of deep feature fusion by combining attention-based DL models with descriptors within a task-oriented framework. Two attention-enhanced CNN architectures, Xception and InceptionResNetV2, were utilised for the classification of Group 1 and Group 2 chest X-ray images, with the attention layers activated to improve feature representation. To further augment the feature space, local descriptors were extracted using the SIFT method.

Each classification task (Group 1 and Group 2) was trained and evaluated independently using separate classifiers in a standard classification setup. The fused feature representations, combining deep-model outputs with SIFT descriptors, were used to train multiple machine learning classifiers, and their performance was systematically evaluated. The top-performing classifiers for each group were selected, with the results summarised in Table 16. The hybrid strategy, which combines attention-based deep features with SIFT descriptors, produced significant accuracy gains, achieving 98.50% for Group 1 and 100% for Group 2. These findings demonstrate that integrating feature extraction with DL substantially improves classification performance, validating the effectiveness of hybrid feature fusion in chest X-ray image analysis.

Experimental evaluation of deep feature fusion for ablation study 3 (Chest X-ray)

During this phase, we established two fusion pools designed to extract features from attention-enhanced models, specifically Xception and InceptionResNetV2, for Group 1 of the chest X-ray dataset. This methodology was similarly applied to Group 2, utilising the same models to maintain consistency across both diagnostic groups. To further enhance the feature representations, we implemented the SIFT, a technique recognised for its ability to detect and describe robust local features accurately. The integration of SIFT significantly improved the feature selection process by emphasising discriminative regions that are critical for classification.

The features derived from the four DL models, Xception_Group1, InceptionResNetV2_Group1, Xception_Group2, and InceptionResNetV2_Group2, were utilised as inputs for training multiple machine learning classifiers across both chest X-ray groups. As demonstrated in Table 17, there was a marked improvement in classification accuracy when applying the same classifier to both groups. Specifically, Group 1 achieved an accuracy of 98.53%, while Group 2 achieved 100% accuracy. These findings underscore the effectiveness of the proposed feature fusion and selection strategy, highlighting its capacity to enhance the performance of machine learning classifiers in chest X-ray classification tasks.

To gain a deeper understanding of how DL models classify chest X-ray images into Group 1 and Group 2, we utilised a range of interpretability techniques, including Grad-CAM, occlusion sensitivity, and t-SNE visualisation. Grad-CAM and Occlusion Sensitivity were employed to identify the specific regions within each image that significantly influenced the model’s predictions. These techniques highlighted the areas on which the models relied most for classification, confirming their focus on clinically relevant structures. The resulting visual explanations are presented in Fig. 4.

Finally, we utilised t-SNE to project high-dimensional feature representations into a two-dimensional space as illustrated in Fig. 5. The resulting t-SNE plot revealed well-separated clusters for both chest X-ray groups, indicating that the model successfully learned distinct, discriminative features for each class. This combination of visualisation techniques enhances the model’s interpretability and fosters trust in its clinical reliability.

Implications

The proposed framework not only achieves high classification accuracy but also significantly enhances interpretability, a crucial factor for clinical adoption. By effectively identifying diagnostically relevant areas within noisy or ambiguous X-ray images, the model serves as a dependable decision-support tool for clinicians. Additionally, its demonstrated generalisability across various medical imaging domains indicates its broad applicability to a wide range of diagnostic tasks. With classification accuracies of 95.48% on MURA X-ray images and 99.80% on chest X-rays, the framework demonstrates considerable promise in supporting radiologists in high-stakes clinical environments, ultimately enhancing both diagnostic confidence and efficiency.

Limitations

The proposed framework, although demonstrating promising results, presents certain limitations. The integration of multiple attention layers within a CNN can introduce practical challenges, such as increased model complexity, heightened computational overhead, and extended training durations. This computational intensity could potentially impede deployment in environments with limited resources, especially where access to high-performance hardware is constrained. Moreover, the framework’s lack of integration with non-imaging data, such as patient history, laboratory results, and clinical notes, restricts its capacity to incorporate essential contextual information, which is vital for making accurate diagnoses and informed decisions in clinical practice.

Future work

Future efforts will focus on optimising the model for enhanced efficiency and smooth integration into clinical workflows. Additionally, validation within real-world settings will be prioritised to ensure the model’s robustness. Subsequent iterations will integrate multimodal data, including laboratory results and patient history, to facilitate more comprehensive clinical decision-making.