Augmenting medical visual question answering with mixup, label smoothing, and layer-wise relevance propagation eXplainable Artificial Intelligence

MVQA datasets

The MVQA datasets used in this research include the generated medical Path-VQA dataset and collected VQA-MED datasets. The Path-VQA dataset consists of pathological VQA queries collected from online digital libraries and pathology textbooks. The pathology images in the Path-VQA dataset are acquired from the generic textbook named Textbook of Pathology using a semi-automated pipeline approach (Mohan, 2010). The QA pairs associated with this dataset were acquired from He et al. (2020). The images and QA pairs are mapped sequentially, and the annotations are reviewed by a medical annotator.

The VQA-MED datasets are collected from ImageCLEF VQA-MED online forum through the tasks conducted from 2018 to 2021. The number of samples of each dataset are given in the original dataset column of Tables 2 and 3. The VQA-MED 2018 and 2019 datasets consists of radiological images related to different organs, planes, abnormalities and modalities are developed from PubMed Central articles and QA pairs from rule-based question generation system (Hasan et al., 2018; Abacha et al., 2019). The VQA-MED 2020 and 2021 datasets consist of abnormality type visual queries related to different organs are acquired from the MedPix and VQA-MED 2019 datasets (Abacha et al., 2020, 2021). These datasets collected from ImageCLEF provides de-identified, ethically released images, ensuring compliance with privacy standards such as Health Insurance Portability and Accountability Act (HIPAA)/General Data Protection Regulation (GDPR). Moreover, to maintain clinical validity, all augmented samples from the proposed work are reviewed by a certified pathologist. This step ensured that transformations did not alter the diagnostic features or introduce unrealistic artifacts. Only clinically appropriate augmentations were retained for training. These measures were taken to ensure that both the original and augmented data are ethically sourced, medically valid, and suitable for model development. From Tables 2, 3, 4 and 5, these MVQA datasets are represented as D1, D2, D3, D4 and D5 for VQA-MED 2018, VQA-MED 2019, VQA-MED 2020, VQA-MED 2021 and Path-VQA datasets respectively.

Augmentation of MVQA datasets

The QA pairs and images of MVQA datasets are improved using proposed augmentation approaches namely, QAI and MII and it is given as Functions 1 and 2 in Algorithm 1.

The mathematical parameters used in Algorithm 1 are: let X be the input consists of both Question-Answer pairs (QAP) and Images (IM) i.e., $X=(X(1),X(2),\dots X(i))$ consists of $QAP=(QAP(1),QAP(2),\dots QAP(i))$ and corresponding $IM=(IM(1),IM(2),\dots IM(i))$. The labels are represented as, $L=(L(1),L(2),\dots L(l))$ and it should satisfy the uniqueness property. This property states that, $\mathrm{\exists }L(i),L(j)\epsilon X$ such that $P(L(i))\wedge P(L(j))$ is true then $L(i)=L(j)$, where $L(i)$ and $L(j)$ represents two different labels.

The MVQA datasets collected from different sources across various years are first normalized to the same format. In this step, images with one QA pair are considered as is, without any modification. However, images with more than one QA pair are split into individual samples, each containing one QA pair and its corresponding image. Next, the QA pairs and medical images in these datasets are improved through a sequence of steps, as discussed in Functions 1 and 2. Finally, the resulting medical images are concatenated with their respective QA pairs to create the improved MVQA datasets.

Function 1: Improvement of QA pairs

In Function 1, the irrelevant QA pairs in the datasets are identified, reduced and augmented using suitable mathematical functions. The three main steps and two conditions used in this function are as follows:

Step 1: Computation of Label-wise QA pairs (LQAP)

Where M and L represents number of QA pairs and Labels respectively.

The amount of QA pairs ( $QAP(m)$) under each Label (L) is computed based on the Eq. (1) as,

(1) $$LQAP=\sum _{l=1}^L\sum _{m=1}^M\frac{QAP(m)}{L(l)}.$$

Step 2: Finding the hardsample threshold value ( $LQA{P}_{HT}$)

From the LQAP, the hardsample threshold value ( $LQA{P}_{HT}$) is computed and it is given in Eq. (2). The $LQA{P}_{HT}$ is required because some of the labels have less number of samples and other labels have relatively elevated number of samples which degrades the overall performance of the system.

(2) $$LQA{P}_{HT}=\mu \sum _{l=1}^{L}LQAP(l),$$where $\mu$ is the average of the total number of samples.

Condition 1

If $LQAP(l)<(LQA{P}_{HT}{)}^{1/3}$, then consider the respective QA pairs as victims and remove them for each label, and the resulting updated datasets are named as $LQA{P}_{UD}$.

Step 3: Selecting appropriate average value ( $LQA{P}_{AV}$)

The average value $LQA{P}_{AV}$ is computed based on $LQA{P}_{UD}$ is given in Eq. (3).

(3) $$LQA{P}_{AV}=\mu \sum _{l=1}^{L_{upd}}\sum _{m=1}^{M}LQAP(l,m).$$

Condition 2

If $LQA{P}_{UD}(l)<LQA{P}_{AV}$ then augment QA pairs till all the labels have equal or more than the average value count. In this, the least frequently used QA pairs within the label gets highest priority.

The improved QA pairs after applying Function 1 is given in Table 2 for all MVQA datasets. In this table, based on the hardsample threshold, the QA pairs are regularized. The average value w.r.t. each label is computed and augmented from the regularized QA pairs.

Function 2: Improvement of medical images

Based on the improved QA pairs, the respective images are removed from $IM(n)$ i.e., $\mathrm{\forall }QAP\notin x$ then remove the respective IM.

The modified images are improved as given in Function 2, a sequence of three steps and two conditions. The three steps are count of the number of images under each label (LIM), hardsample threshold value ( $LI{M}_{HT}$) and average value ( $LI{M}_{AV}$). The computation of these steps are similar to Eqs. (1), (2) and (3), respectively. The two conditions are,

Condition 3

If LIM of the particular label is less than the one-third of hardsample threshold value then consider the respective image as victim and remove it from each dataset and named it as $LI{M}_{UD}$.

Condition 4

If $LI{M}_{UD}<LI{M}_{AV}$ then augment QA pairs using mixup and label smoothing as discussed in the article (Sheerin & Kavitha, 2022).

The label smoothing adjusts the probability of the target samples to remove the hard samples. The mixup technique augments the image by creating the new image $\hat{x}$ and $\hat{y}$ from two samples using linear interpolation. The improved medical images before and applying Function 2 is given in Table 3.

The output of Functions 1 and 2 are combined based on the ImageID in Algorithm 1 and it is named as Improved dataset.

MVQA model creation

The proposed MVQA framework is implemented using two distinct deep learning architectures, both integrating visual and textual modalities. In Approach 1, image features are extracted using a pre-trained VGGNet model (Simonyan & Zisserman, 2015), wherein the final fully connected layer is frozen to preserve learned high-level visual representations while preventing overfitting on limited medical data. These image features are passed to a LSTM network (Hochreiter & Schmidhuber, 1997) to process sequential text inputs (i.e., questions) and extract semantic embeddings. The output vectors from both modalities are concatenated and fed into a classification layer to predict the corresponding answer. The answers span categories such as anatomical organ, imaging plane, pathological abnormality, or modality type (Li et al., 2023).

In Approach 2, the image feature extraction is conducted using a ResNet architecture (Wu, Shen & Hengel, 2019), with the final fully connected layer also frozen to retain pre-trained weights. Similar to Approach 1, the LSTM network encodes the textual component of the question. The visual and textual embeddings are then concatenated and projected onto a fully connected layer, with the output dimension matching the total number of class labels across all defined categories. Notably, both approaches begin with a dataset-specific preprocessing pipeline, detailed in Eq. (4), which standardizes and augments the data to enhance generalizability and reduce domain-specific variance. This preprocessing may involve resizing, normalization, and tokenization strategies tailored for the medical imaging domain.

(4) $$PID=\sum _{i=1}^{I}PP(ID(i)),$$where $PP(ID(i))$ represents the pre-processing of each samples of the improved datasets, namely colour conversion to RGB, size adjustment, expand dimension, etc., whenever required. The Proposed MVQA (PMV QA) model generated from the Processed Improved Dataset (PID) is used to predict answer and it is named as $PMVQ{A}_{PA}$ in Eq. (5).

(5) $$PMVQ{A}_{PA}=\sum _{i=1}^{I}PMVQA(PID(i)).$$

Performance analysis

The performance analysis of the MVQA model is evaluated using quantitative metrics and LRP XAI visualization.

Analysis of MVQA results using quantitative metrics

The results of the MVQA models generated from Approach 1 and Approach 2 are validated using quantitative metrics and compared with state-of-arts (SoA). The performance analysis using accuracy and BLEU score for all datasets are given in Table 4 for approach 1 and Table 5 for approach 2. In these tables, SoA1, SoA2, SoA3, SoA4 and SoA5 represent Peng, Liu & Rosen (2018), Khare et al. (2021), Joshi, Mitra & Bose (2023), Gong et al. (2021) and Naseem, Khushi & Kim (2022) respectively. These two approaches are compared for all improved datasets in Fig. 2 for better understanding. From Table 4, it is observed that the improved VQA-MED 2019 and Path-VQA datasets attained 12.1% and 2.6% increased accuracy than SoA2 and SoA5 respectively. Because the VQA-MED 2019 dataset takes differentially weighting to each part of the sample images and in Path-VQA dataset, most of the labels have more than one sample hence performance is maintained. In terms of BLEU score, the VQA-MED 2018 and 2019 datasets achieved 22.3% and 9.3% increased BLEU score than SoA1 and SoA2 respectively. Because the VQA-MED 2018 dataset maintained the least sample to label ratio. The overall inference from Table 4 is that the accuracy and BLEU score are improved from 1.5% to 11.5% and 0.7% to 12.4% respectively for improved datasets.

From Table 5, it is inferred that the VQA-MED 2018, VQA-MED 2019, and Path-VQA datasets perform better than SoA1, SoA2, and SoA5 by 23.5%, 11.8%, and 3.4%, respectively, when using Approach 2 in terms of BLEU score. Overall, the performance improvement ranges from 3% to 12.6% in accuracy and from 1.3% to 11.0% in BLEU score for the improved datasets. This demonstrates that normalization through mathematical computation helps standardize the datasets and supports the generation of more effective models by augmenting underrepresented samples. Figure 2 shows that Approach 2 (ResNet followed by LSTM) outperforms Approach 1 (VGGNet followed by LSTM). Approach 2 is also visualized using XAI, which highlights the most significant information in the input data.

To assess the robustness of the observed performance improvements, statistical significance testing using a paired t-test between models trained on the original and augmented datasets are conducted. The results indicate that the improvements in both accuracy and BLEU score are statistically significant across all datasets, with p-values <0.05. In addition, we report 95% confidence intervals for each metric, calculated over five independent training runs with different random seeds. These intervals demonstrate consistent gains and minimal variance, further confirming the reliability of the proposed augmentation techniques. Detailed results, including mean $\pm$ standard deviation and p-values, are given in Table 6.

Analysis of visual representation using XAI

The most significant region in the image and question, which contributes for answer prediction are highlighted using XAI techniques and it is shown in Figs. 3, 4, 5, 6 and 7. From Figs. 3, 4 and 5, the MVQA datasets are analysed under different perspectives namely, (i) Visualization of one image with multiple QA pairs (ii) Visualization of one QA pair with multiple images (iii) Visualizing the QA pair for the sample from original and improved dataset. Since Path-VQA dataset are less complex, VQA-MED 2021 dataset are analysed by visualizing different abnormalities of same body region and it is shown in Fig. 6. The VQA-MED 2021 dataset is chosen because it consists of abnormality related questions which are difficult to answer as compared with other types. Among the images, the samples related to abnormality on and around the knee joint are considered for understanding the disparity among similar type medical images. Finally, the limitations with respect to each MVQA dataset is shown in Fig. 7.

In Fig. 3, two different images of same question “What is there in the image?” are illustrated with answers, heatmaps and super imposed images. From the figure, it has been inferred that the generated heatmap varied by colour intensity, location and quantity of the image depending on the input and, as a result the respective super imposed image also varied accordingly. Similarly in Fig. 4, four different QA pairs for the answers “affected area on right”, “nuclei”, “outlines of tubules” and “granular debris” are illustrated for the same medical image. From the figure, it has been inferred that depending on the QA pairs of the same image, the respective heatmap varied because of the contribution of different regions in the image towards answer prediction and it is reflected in the generated super imposed image.

The XAI visualization and its analysis for QA pairs are shown in Fig. 5. In this figure, the answer to the question “What does the mucosal surface show?” is “tumour in lumen,” according to the dataset. The model trained on the original dataset predicts the answer as “tumour” (partially correct), whereas the model trained on the improved dataset correctly predicts “tumour in lumen.” In this figure, the context of the words is represented using three different colors: green (positive context), white (neutral context), and red (negative context). In Fig. 5B, the context of the keywords “does,” “mucosal,” and “surface” contributes to the answer prediction from different perspectives and is shown in green. However, in Fig. 5A, the context of the answer with respect to the question varies from positive to negative. So far, the importance of LRP XAI in interpreting pathology images and QA pairs has been demonstrated through these visualizations. The role of XAI in identifying the reasons behind misclassified samples is discussed in the following paragraph.

The MVQA (before and after improvement) outcome and LRP XAI output are shown to demonstrate the effect of augmentation and visualisation in answer prediction in Fig. 6. In Figs. 6A, 6B and 6C represents correctly classified samples, Figs. 6D and 6E represents wrongly classified samples. Among the correctly classified samples, the proposed MVQA model predict appropriate answer after dataset improvement that shows the effect of augmented datasets in developed MVQA model. In terms of visualization, LRP XAI highlights the region around the knee joint (a) “Osteochondroma” is an overgrowth of cartilage at the knee joint (b) “Osteosarcoma” is a bone cancer that is developed in the osteoblast cell and hence the long bone is highlighted (c) “Radial head fracture” occurs at the diverging line from the joint, so the region around radial head is highlighted. On the other hand, for (d) Instead of “Chondrocalcinosis” (Calcium pyrophosphate crystal deposits in the joint tissues), the model predicted the output as “Osteosarcoma” (e) Model predicted output as “fat necrosis” (death of fat tissues due to injury) instead of “Osteochondroma”. As per the visualization in both cases, instead of knee joint, the model learned the random features from surrounding bones and muscles. As per the visualization in both cases, instead of knee joint, the model learned the random features from surrounding bones and muscles. Moreover, the limitations corresponding to different MVQA datasets are also visualized using LRP XAI.

The limitations of all MVQA datasets are also analysed using LRP XAI are shown in Fig. 7 for some samples. In this figure, the output are shown in the order of the input images, question with actual answers (QA pairs), wrongly predicted answers and the output of LRP XAI visualization. Because these samples have more visualization effect with less deviation in interpretation for different organs, planes and modalities. For the sample of VQA-MED 2018 dataset, the proposed model captured the features from the lymph region instead of the lung and hence, the lymph node related abnormality is generated and it is visualized in LRP XAI output. In the case of VQA-MED 2019 dataset, the region in the patellofemoral bone is identified but the plane type is varied from sagittal to longitudinal because these two plane types are slightly varied from each other. In the VQA-MED 2020 dataset, the proposed model considers the Phalanges aspatellofemoral bone and hence predicted output as osteochondritis dissecans. For the VQA-MED 2021 dataset, the answer is predicted based on the growth of inflammatory cells because of the synonyms used in the question. In Path-VQA dataset, the tissue is damaged between portal tract and periportal parenchyma but the model captured the region at the outer layer and hence output as peripheral.