Towards trustworthy digital health: reliability of uncertainty-aware convolutional neural networks

The role of AI in improving diagnostic processes

Machine learning (ML) and deep learning (DL) can analyze vast amounts of medical data, including images, patient records, and genetic information, to assist healthcare professionals in identifying diseases and conditions with greater precision. These technologies excel in image-based diagnostics, where AI models can detect patterns in medical imaging data, such as X-rays, MRIs, or pathology slides, that may be imperceptible to the human eye. In addition to improving diagnostic accuracy, AI enables early detection of diseases, personalized treatment plans, and reduced human error. As AI continues to evolve, its role in healthcare is expected to grow, offering the potential to revolutionize diagnostic processes and enhance patient outcomes across various medical specialties.

Figure 1 reports images related to Melanoma and Diabetic Retinopathy, two diseases used in this work as a real CAD scenario.

The rising need for trustworthiness in AI in digital health

The high and increasing accuracy, together with the growth of AI in the Digital Health field, leads to a growing interest in determining how reliable the answers to these methods are from a clinical point of view. Recent studies highlight the growing interest of the scientific community in trust in AI systems in the healthcare domain, increasingly considering it a key factor for clinical adoption, on par with technical performance. In particular, Hassija et al. (2024) emphasize that AI must be implemented reliably and transparently for trust and ethical clinical decisions.

Helenason et al. (2024) investigated the feasibility of an AI-based CDSS to detect cutaneous melanoma in primary care. The results demonstrated that an AI-based CDSS could play an important role in cutaneous melanoma diagnostics due to sufficient evidence of diagnostic accuracy. However, the reports from the interviews with fifteen primary care physicians revealed that trust in the CDSS emerged as a central concern. Interestingly, scientific evidence supporting the diagnostic accuracy of the CDSS was identified as a key factor that could increase trust.

Fosso Wamba & Queiroz (2023) highlight the growing role of responsible AI⁹ in digital health, stressing the need for trustworthiness and ethical integrity. Privacy concerns, ethical issues, and cultural resistance are key barriers to AI adoption. Responsible AI approaches help address medical tensions, particularly in data privacy and governance. Publication trends and emerging technologies further support the necessity of ethical and high-performing AI systems (Reinhardt, 2023).

According to Laux, Wachter & Mittelstadt (2024), due to the black-box nature of AI models, patient and physician confidence in diagnosis and treatment is essential in every guided AI-clinical diagnosis; therefore, tackling the lack of transparency in these AI models becomes essential for researchers to encourage ongoing collaborations between AI developers and clinical end users. The author’s conclusion highlights the importance of transparency and participatory approaches in building trust, especially in sectors like healthcare. Also, the authors highlight how trust research remains fragmented and how further research and sector-specific regulation are necessary to address the complexities of trust in AI, ensuring that institutions and intermediaries play a key role in fostering trust and transparency in AI-driven clinical decisions.

In this evolving scenario, ethical considerations, including consent, explainability, and fairness, have played a central role in designing AI systems, effectively becoming key aspects to be considered “by design” in their development and implementation. For example, future AI models in Europe must follow the “do not harm” rule (https://artificialintelligenceact.eu/article/5/) and align with the EU Parliament’s recommendation. Additionally, they must adhere to principles of transparency¹⁰ , ensuring that AI systems are transparent and their decision-making processes are understandable to users. Fairness must also be a central consideration, ensuring that AI models “do not harm” (discriminate) and operate equitably across different populations. These requirements aim to promote trust, accountability, and ethical responsibility in AI systems, aligning with broader legal and ethical standards that govern their deployment in various sectors, including healthcare, to safeguard public well-being. If we agree with the previous observations and results, it is reasonable to state that guaranteeing interpretability, explainability, and trustworthiness will be essential for future AI systems in healthcare, alongside other key aspects such as performance, robustness, and ethical considerations. Therefore, this contribution aims to stimulate discussion on the need for a quantitative definition of “trustworthiness”. We argue that this indicator should be defined to support the development of all future healthcare AI-based applications.

Consequently, we propose a raw definition of this Trustworthiness Indicator ( $\mathrm{\Phi }$), applied to two specific CAD scenarios: the Melanoma Binary Classification Problem (MBCP), and to Diabetic Retinopathy Classification and Grading (DRCG), introduced in brief in the following sections. These problems help us investigate ( $\mathrm{\Phi }$) properties in scenarios that cover image processing aspects of the AI-CAD system in healthcare.

Finally, we discuss future works and the limitations of our current proposal. In particular, we highlight the need to understand whether this indicator should be used alongside classical performance metrics, such as accuracy and precision, or whether it could serve as a stand-alone indicator, which requires deeper investigation. Moreover, beyond images, future work should also consider the importance of purely numerical analyses, as they could provide valuable insights in complement to visual data.

Explainability, interpretability (XAI), and uncertainty in AI for healthcare

Decisions in healthcare carry ethical and psychological implications. They could profoundly impact patient well-being and life because they can determine the accuracy of diagnoses and the effectiveness of treatments¹¹ . Timely decision-making is also critical, as delays can worsen conditions, reduce survival chances, and compromise the overall safety of medical interventions. Therefore, if we want to use AI in digital health, it is reasonable to consider it crucial to ensure that AI systems are transparent, interpretable, explainable, and ethically designed to support accurate diagnoses, effective treatments, and patient safety while maintaining trust and minimizing potential risks.

Interpretability refers to how easily a human can understand the internal workings of an AI model¹² . It focuses on how inputs relate to outputs, even if the reasoning process is not explicitly stated. It explains why a model made a specific decision. For example, Grad-CAM highlights which part of an image influenced the classification of a CNN (Selvaraju et al., 2020).

Explainability refers to the degree to which AI systems can clearly articulate their decision-making processes. The AI system can provide understandable reasons for its decisions, often involving the generation of human-friendly explanations for complex or ‘black-box’ models like deep learning networks.

Unfortunately, defining trust is challenging. Consider a CNN model used to classify medical images. If Grad-CAM highlights irrelevant regions while the model still achieves high accuracy, the model is interpretable but not trustworthy. On the other hand, consider a CNN that consistently provides accurate and well-calibrated diagnoses but functions as a “black box” without clear explanations. In this case, the model is trustworthy but not interpretable.

According to Starke & Ienca (2024), many forms of trust in AI exist, and the deep learning-based applications preclude comprehensive human understanding. Nevertheless, trustworthiness refers to how reliable, robust, and safe the AI system is across different conditions. It assesses whether a model’s outputs can be depended upon. For example, a well-calibrated confidence score (e.g., via temperature scaling) ensures that when the model predicts “90% confidence,” the actual correctness is close to 90%.

Table 1 summarises the distinction between XAI and trustworthiness. XAI focuses on making AI decisions understandable, while trustworthiness ensures robustness and reliability essential for clinical adoption.

If we integrate AI into the healthcare field and accept that decisions in healthcare can profoundly impact patient outcomes, healthcare providers and patients must understand how AI arrives at its recommendations. If an AI system functions as a “black box’ offering little insight into the rationale behind its decisions, skepticism and hesitation in adopting the technology are likely: this is particularly concerning in healthcare, where the stakes are high, and errors can have severe consequences.

According to Gille, Jobin & Ienca (2020), clinicians need confidence in their ability to interpret AI-generated insights effectively (and they must not see AI as concurrent), while patients require clear explanations of treatment options influenced by AI and, to earn trust, healthcare AI must undergo rigorous validation and testing to ensure accuracy, safety, and performance across diverse populations and medical conditions. Additionally, according to Kundu (2021), healthcare AI must be explainable, and its explanations must have clinical relevance.

As reported by Saraswat et al. (2022), the XAI research field provides multiple techniques to help understand the decision-making processes of complex artificial intelligence models, providing insights into the factors contributing to their outputs, allowing users to verify and validate the reasoning behind AI decisions: this aspect may lead to increased trustworthiness in AI system in digital health. Also, according to Hassija et al. (2024), XAI can make AI more accessible, reliable, and trustworthy in clinical settings.

Aims and scope of the work

Based on the previous facts, it is reasonable to conclude that a “fast, robust, and reliable” diagnosis system (human or AI) can improve patients’ well-being. Looking at the literature, we can discover that deep learning boosted the proposal of tools for assisting in the early identification of melanoma: starting in 2016, the scientific community produced and accepted more than 450 proposals¹³ describing melanoma and generic skin lesions CAD working on the clinical, dermoscopic, and histological image. Also, more than 600 accepted contributions¹⁴ related to diabetic retinopathy were accepted in the same period.

Agreeing with the observation reported in the previous section, it is reasonable to state that we can design “fast” and “robust” systems by using AIms. We can also design some “explainable” systems with AI and XAI. Now, we still must complete our journey of integrating “reliability” and “trustworthiness” into our system, surpassing the standard XAI techniques such as Grad-CAM, SHAP, and LIME¹⁵ .

We propose the Trustworthiness Indicator ( $\mathrm{\Phi }$) as a metric focused on assessing model robustness through internal activation pattern consistency, complementing existing UQ techniques. Unlike traditional UQ techniques that primarily focus on predictive uncertainty, ( $\mathrm{\Phi }$) assesses model robustness by evaluating the consistency of internal activation patterns across different CNNs. We hypothesize that activation pattern consistency among diverse models and high consensus in prediction output may indicate model robustness and generalizability, though further research is needed to determine the extent of this relationship.

While further empirical validation is required, we propose ( $\mathrm{\Phi }$) as an additional metric that can be used alongside standard performance indicators rather than replacing existing approaches. This combined approach can improve alignment between numerical reliability metrics and human-understandable explanations, enhancing clinicians’ confidence in model predictions. However, broader institutional trust in AI depends on regulatory validation, real-world performance, and user experience. Thus, additional validation and comparative analysis are essential to assess ( $\mathrm{\Phi }$)’s practical impact (Helenason et al., 2024).

In particular, this work assumes the working hypothesis (WH): “as many activation patterns are shared across multiple models and as high are network accuracies across these models, the more we can rely on these model sets”. ( $\mathrm{\Phi }$) shall indicate the strength of the local feature-level consensus across CNNs.

Grad-CAM

Gradient-weighted Class Activation Mapping (Grad-CAM) is a visualization technique for interpreting convolutional neural networks (CNNs). It generates class-discriminative heatmaps that highlight the important regions of an input image influencing the network’s decision. Grad-CAM computes importance scores for each feature map in a convolutional layer by utilizing the gradients of the target class score with respect to the feature maps. Additionally, Grad-CAM produces heatmaps overlaid on input images, emphasizing the regions that the CNN relies on for classification. These visualizations enhance model interpretability and facilitate debugging (Selvaraju et al., 2020).

Figures 2 and 3 show examples of Grad-CAM outputs, illustrating which parts of the images are most relevant for the model’s classification decisions.

Structural similarity index measure

The Structural Similarity Index (SSIM) is a metric for evaluating the visual similarity between two images (Brunet, Vrscay & Wang, 2011). Unlike traditional pixel-based metrics like Mean Squared Error (MSE), which measure only pixel value differences, SSIM considers structural information, luminance, and contrast, providing a more perceptually relevant assessment of image quality. By comparing local patterns of pixel intensities and their spatial relationships, SSIM closely aligns with human visual perception. It outputs a value between −1 and 1, where 1 indicates perfect similarity between two images, and lower values signify perfect dissimilarity (anticorrelation). SSIM is widely used in image compression, restoration, and computer vision because it captures perceptual differences.

Performance metrics

We computed the well-established classification metrics to analyze the behavior of $\mathrm{\Phi }(.)$. We then compared $\mathrm{\Phi }(.)$ with these metrics to gain insights into its strengths and weaknesses across various classification scenarios.

We report standard metrics to evaluate model performance in both binary and multiclass settings (Tables 2 and 3). For binary classification (melanoma detection), we compute Accuracy (ACC), Sensitivity (SN), Specificity (SP), Precision, and F1-score (Table 2). For multiclass classification (diabetic retinopathy grading), we compute overall Accuracy along with macro- and micro-averaged Precision, Recall, and F1-score. Macro-averaging treats each class equally, while micro-averaging reflects instance frequency. We also include Specificity, False Positive Rate (FPR), False Negative Rate (FNR), and Hamming Loss. Hamming Loss quantifies the fraction of misclassified labels, with 0 indicating perfect classification (Table 3).

Image improvements and segmentation

Due to the limited quality of the images, which are affected by poor contrast and artifacts, three different preprocessing steps were applied in this work to improve feature extraction during network training: Contrast Enhancement, Hole Erosion, and Image Segmentation. This section provides a brief introduction to these techniques.

DATASETS

Selection method

The selection of the techniques implemented in this work was guided by their relevance to the medical image classification tasks addressed, their support for model interpretability, and their potential to contribute to the assessment of trustworthiness in AI-based diagnostic systems.

CNN were chosen as the primary models due to their established effectiveness in tasks such as melanoma detection and DR grading. A diverse set of architectures was included, ranging from lightweight to deeper models, to capture a broad spectrum of learning behaviors and assess the consistency of internal feature representations across different network types.

Grad-CAM was selected as the interpretability method because it enables the visualization of class-discriminative regions in input images, allowing clinicians to inspect and validate the reasoning behind model predictions. This choice aligns with the growing demand for explainability in AI applications within the healthcare sector.

The Trustworthiness Indicator proposed in this study was specifically designed to measure the consistency of activation patterns across models based on the hypothesis that high agreement among independently trained models may signal robust and generalizable decision processes.

The selection of public datasets—ISIC for skin lesion classification and EyePACS for retinal disease grading—was driven by their clinical relevance, size, and widespread use in benchmarking AI models, ensuring that the evaluation would be grounded in real-world scenarios.

Computing infrastructure

The experiments were conducted on a workstation running Windows 11 version 24H2 (build 26100.4061), equipped with an AMD Ryzen Threadripper PRO 5955WX 16-core processor (32 threads, 4 GHz), 128 GB of RAM, and a single NVIDIA RTX A4000 GPU with 16 GB of VRAM. The implementation was developed using MATLAB R2023b with the Parallel Computing Toolbox and Deep Learning Toolbox.

Image pre-processing

For the ISIC dataset, we use a Sobel-based edge segmentation pipeline combined with morphological refinements. We first applied k-means clustering with two clusters, followed by hole filling. We converted the image to grayscale, and an automatic Sobel threshold was computed; this value was then scaled by a fudge factor of 0.5 to obtain a more conservative edge map. To connect fragmented edges, we applied dilation with linear structuring elements of size 5 (90^∘) and 3 (0^∘). Subsequently, residual holes were filled, border-touching artifacts were removed using 4-connectivity, and two successive erosions with a diamond structuring element of radius 2 were performed. The final mask was applied to the original image to isolate the region of interest. No watershed transform was employed in this workflow.

For the EyePACS dataset, we enhanced image contrast in the luminance channel using Contrast Limited Adaptive Histogram Equalization (CLAHE). Images were converted to the CIE-Lab color space, the L channel was normalized, processed with adapthisteq using MATLAB default parameters, and then reconverted to RGB. This approach enhanced the visibility of relevant structures while minimizing the amplification of noise. No additional denoising, class rebalancing, or label-noise handling was applied at this stage, as the aim of the study was not to optimize predictive accuracy but rather to evaluate the behavior of the proposed reliability indicator under realistic conditions.

For reproducibility, the exact MATLAB scripts used for these steps are made available in the project repository.

Network preparation

To prepare the networks, a specific configuration of the AlexNet architecture was designed for training from scratch. In this version, the network is initialized without pre-trained weights, so all parameters start from random values, and the entire model is trained from the beginning. The input layer is adapted to match the specified INPUT_SIZE, ensuring compatibility with the input data. The final fully connected layer (fc8) is replaced to produce many outputs equal to OUTPUT_CLASSES, corresponding to the number of target classes in the classification task (2 for melanoma and 5 for DR).

Training setup

In our experimentation, two distinct training configurations were established to evaluate the impact of data augmentation on the performance of a neural network. The first configuration involved standard training without any augmentation, while the second introduced various random transformations to the training images aimed at increasing dataset diversity and enhancing the model’s ability to generalize. Data augmentation techniques included random horizontal reflections simulating mirrored images to introduce orientation variability. In addition, random translations were applied along the horizontal and vertical axes, allowing shifts within a range of −180 to 180 pixels. Images were randomly scaled in both dimensions, with scaling factors sampled between 1 and 30. These transformations were selected to mimic a broad spectrum of real-world variations in object positioning, size, and orientation, thereby increasing the robustness of the model.

Both configurations employed the same optimization strategy, based on stochastic gradient descent with momentum (SGDM), using an initial learning rate of 0.0001, a maximum of 20 training epochs, and mini-batches of 50 samples. The dataset was shuffled during training at the beginning of each epoch, and validation was performed every 100 iterations. The training process was monitored through real-time visual feedback, while intermediate textual outputs were suppressed. In order to ensure computational efficiency, training was executed using GPU acceleration; however, the experimentation is reproducible using CPUs. The only difference between the two setups lies in the validation dataset: the first configuration used the original, unaugmented validation set, while the second employed an augmented version of the validation data, incorporating the same transformation strategies used during training. This controlled setup directly compares model performance under conditions with and without data augmentation.

Training starts by selecting the training dataset from a predefined split, in which the entire dataset has been partitioned into training, validation, and test sets according to a specified ratio. A string identifier representing the split configuration was generated and used to organize the results of each experiment. This identifier dynamically created a structured directory hierarchy to store outputs, including the dataset name, model name, and split configuration: this choice ensured that all experiment results were stored in uniquely identifiable and reproducible locations. The model was then trained using the designated architecture, the selected training data, and the defined training parameters.

Performance evaluation and logging

The resulting model and its performance metrics were saved, including a graphical representation of the training progress, enabling post-hoc analysis of learning dynamics. The evaluation phase continued on the test set, where the model’s classification performance was quantitatively assessed. In addition, visual interpretability techniques—specifically Grad-CAM—were applied to generate saliency maps that highlight the regions of input images most influential in the model’s predictions. All results were systematically stored in a structured directory, organized by dataset and model identifiers, and computations were performed using GPU acceleration where applicable.

In order to evaluate the model’s generalization capabilities, a standardized procedure was followed to test performance on all three dataset partitions: training, validation, and testing. The evaluation process involved classifying each partition and comparing the predicted labels to the corresponding ground truth. The classification accuracy for each set was computed and logged, along with relevant metadata such as the dataset name, the network configuration, and the data split ratios used during training. Results were saved in a tab-separated values file to facilitate subsequent analysis.

In addition to accuracy metrics, confusion matrices were generated for each data partition to provide a more detailed view of classification performance across individual classes. These matrices were saved in plain text format, and corresponding visual charts were exported as image files. The evaluation pipeline was designed for repeatability and transparency, ensuring that each experiment produced a complete and traceable set of outputs within a clearly defined directory structure.

Gradcam extraction and handling

Grad-CAM generates visual explanations of network predictions by producing heatmaps highlighting image regions most influential to the model’s decisions. In our setup, this process takes the dataset name, network name, data split configuration, and test set as input, establishing a structured directory layout based on these parameters and the predicted class labels. Each result consists of a heatmap, produced using the gradCAM function, superimposed on the original image to form an interpretable visualization. These composite images are saved in class-specific folders in .png format to facilitate visual inspection, automatic similarity structure analysis, and comparative evaluation across test samples.

To formalize this step, we denote $I_i$ as the $i$-th image in the test set TS, and $N_x$ as the $x$-th network in the set of trained-from-scratch models TNN. The scoreMap, as defined in (https://it.mathworks.com/help/deeplearning/ref/gradcam.html), encodes regions with high activation values that contributed most to the prediction made by $N_x$. For each $N_x\in TNN$ and $I_i\in TS$, we compute the pair $(S{M}_x(i),G{N}_x(i))$, where $S{M}_x(i)$ is the scoreMap, and $G{N}_x(i)$ is the resulting composite image obtained by overlaying $S{M}_x(i)$ on $I_i$ using the jet colormap (https://it.mathworks.com/help/matlab/ref/jet.html).

Trustworthiness definition

In order to link how differently each network uses different features to make choices with our WH, we defined a Pair Consensus Indicator ( $\mathrm{\Lambda }$). In this work, $\mathrm{\Lambda }\in [0,1]$ and a value near 1 indicates max consensus.

Specifically, we used a composition of the function $\omega (.)$ that integrates the SSIM to compute the similarity between two generic images, as explained in the equation below (Eq. (2)).

(1) $$\omega (x)=\{\begin{array}{cc}0,\hfill & \mathrm{i}\mathrm{f}x<0\hfill \\ x,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}$$

(2) $$\mathrm{\Lambda }(a,b)=\omega \circ SSIM(a,b).$$

SSIM takes values between −1 and 1, with 1 indicating max correlation and −1 max anti-correlation. Therefore, $\mathrm{\Lambda }(a,b)$ takes value on $[0,1]$.

If we consider two generic neural network models that work with images, $N_x$ and $N_y$, and an image $I_i$, we can define:

(3) $$\delta (I_i,N_x,N_y)=\mathrm{\Lambda }(S{M}_x(i),S{M}_y(i)).$$as the correlation-distance between activation features used by network $N_x$ and $N_y$ to choose an answer (the class predicted) $C_x$ and $C_y$, regarding the image $I_i$. In the formula, $S{M}_x(i)$ and $S{M}_y(i)$ denote the Grad-CAM score maps for $N_x$ and $N_y$ on image $I_i$.

Therefore, $C_i$ is the prediction of $N_i$ regarding image $I_i$. This correlation distance is what we define as consensus on feature selections between two different NN models. As a result of the corrected ssim property, $\delta (.,.,.)$ takes on a positive and increasing value the higher the correlation between activation features is, and therefore, the higher the consensus between feature choices.

However, at this point, it is important to note that activation pattern similarity does not necessarily directly correlate with trustworthiness. NN can share activations even when making incorrect predictions, particularly in overfitted models. It is mandatory to consider that a high value of $\delta (.,.,.)$ only indicates a high consensus about features used to make a decision (networks use similar features to make a prediction). Therefore, we have to take into account wrong decisions.

If $C_i$ denotes the ground-truth class for image $I_i$, and $C_x$ and $C_y$ are the labels predicted by $N_x$ and $N_y$, respectively, we define the choice-level network consensus as follows.

Choice-level network consensus and label mapping. Let $p^{(N_x)}(I_i)\in [0,1{]}^K$ denote the softmax probability vector predicted by network $N_x$ for image $I_i$. The corresponding predicted label is

(4) $$C_x=\arg \underset{k\in \{1,\dots ,K\}}{max}{p}_k^{(N_x)}(I_i).$$

An analogous definition holds for $C_y$ with $N_y$. In the binary case ( $K=2$), this is equivalent to thresholding the positive-class probability (or logit) at $0.5$. We denote by $C_i$ the ground-truth class of image $I_i$.

Let $C_i$ be the ground-truth class of image $I_i$, and let $C_x$ and $C_y$ be the labels predicted by networks $N_x$ and $N_y$, respectively. We define the choice-level network consensus as

(5) $$\eta (N_x,N_y,I_i)=\{\begin{array}{cc}1,\hfill & \mathrm{i}\mathrm{f}(C_y=C_x)=C_i,\hfill \\ -1,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}$$

Therefore, given a dataset $D=\{D_1,D_2,\dots D_n\}$ composed of $n$ images and two NN $N_x$ and $N_y$, we can compute what we defines as Local Trustworthiness of a network pair against D as follow:

(6) $$\mathrm{\Psi }(N_x,N_y,D)=\sum \frac{\delta (I_i,N_x,N_y)-\eta (N_x,N_y,I_i)}{n}\mathrm{\forall }{I}_i\in D.$$

Due to the presence of the penalty in $\eta$, please note that $\mathrm{\Psi }(.,.,.)$ take values in [−1, 1].

If we consider a set of $TNN=\{N_1,N_2,\dots N_m\}$ and P(TNN) as the set of all possible pairs within TNN, we can generalize $\mathrm{\Psi }(.,.,.)$ in Trustworthiness of a CNN set against the classification problem on D as follow:

(7) $$\mathrm{\Phi }(TNN,D)=\sum \frac{\mathrm{\Psi }(N_i,N_j,D)}{|P(TNN)|}\mathrm{\forall }(N_i,N_j)\in P(TNN).$$

Also, as defined, it is possible to compute $\mathrm{\Phi }$ against a specific prediction class $C_c$ (for example, benign or melanoma or severe or mild retinopathy).

In that case, the dataset comprises images of the same class $c$, defined as $D_c$.

(8) $$\mathrm{\Phi }(TNN,D_c)=\sum \frac{\mathrm{\Psi }(N_i,N_j,D_c)}{|P(TNN)|}\mathrm{\forall }(N_i,N_j)\in P(TNN).$$

Correlation between the trustworthiness indicator Φ and classical performance metrics

We investigated whether a statistical relationship exists between $\mathrm{\Phi }$ and classical performance indicators. Preliminary correlation analyses suggest that the relationship between $\mathrm{\Phi }$ and metrics such as accuracy, precision, and F1-score is weak or nonlinear. In some cases, local positive trends can be observed—e.g., moderate $\mathrm{\Phi }$ values tend to co-occur with balanced precision and recall, but these trends break down when outlier architectures (e.g., those with overfitting or underfitting behavior) are included.

To assess the statistical relationship between $\mathrm{\Phi }$ and traditional performance indicators, we first computed for each network $N_i$ the mean pairwise trustworthiness on the malignant melanoma class:

$${\overline{\mathrm{\Phi }}}_i=\frac{1}{M-1}\sum _{\genfrac{}{}{0ex}{}{j=1}{j\ne i}}^M\mathrm{\Psi }(N_i,N_j),$$where $\mathrm{\Psi }(N_i,N_j)$ are the values reported in Tables 6 and 7. We then assembled vectors of classical metrics, for example

$$\mathbf{A}\mathbf{C}\mathbf{C}=[(N_1),(N_2),\dots ,(N_M)],\mathbf{F}\mathbf{1}=[1(N_1),1(N_2),\dots ,1(N_M)],$$using the results in Table 5.

Pearson’s correlation coefficient $r$ and Spearman’s rank correlation $\rho$ were then computed between $\{{\overline{\mathrm{\Phi }}}_i\}$ and each performance vector. In all cases, $|r|<0.25$ and $|\rho |<0.20$, corresponding to $R^2<0.06$ for every linear regression. Further, standard transformations (logit, arcsine of proportions, and second-order polynomial fits) did not increase the best $R^2$ above 0.10. These results indicate that $\mathrm{\Phi }$ captures an internal model-agreement dimension largely orthogonal to classical accuracy-based metrics.

Given the bounded nature of $\mathrm{\Phi }\in [-1,1]$, and the inherently bounded classical metrics $\in [0,1]$, we also considered whether the relationship could be linearized through transformations such as logit, arcsin, or polynomial regression. However, even under these transformations, the overall coefficient of determination ( $R^2$) remained low across all tests: this reinforces the hypothesis that $\mathrm{\Phi }$ and classical metrics capture distinct dimensions of model evaluation.

Notably, the malignant class consistently exhibits lower and more variable $\mathrm{\Phi }$ values across all model pairs: this may reflect the inherent ambiguity or variability in pathological patterns associated with malignant melanoma, leading to divergent internal representations even among well-performing classifiers. Such discrepancies could be clinically significant, indicating reduced model consensus in high-risk cases.

The statement regarding the bounded nature of $\mathrm{\Phi }$ and the failure of standard linearizing transforms is grounded in our quantitative correlation experiments. We first extracted each network’s mean pairwise trustworthiness on the malignant melanoma class:

$${\overline{\mathrm{\Phi }}}_i=\frac{1}{M-1}\sum _{\genfrac{}{}{0ex}{}{j=1}{j\ne i}}^M\mathrm{\Psi }(N_i,N_j,D_{\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{a}\mathrm{n}\mathrm{t}}),$$

Using the values in the trust matrix (Table 6 for benign and the corresponding matrix for malignant). Classical performance vectors, for example

$$\mathbf{A}\mathbf{C}\mathbf{C}=[(N_1),\dots ,(N_M)],\mathbf{F}\mathbf{1}=[1(N_1),\dots ,1(N_M)],$$were taken directly from Table 5 Computing Pearson’s $r$ and Spearman’s $\rho$ between $\{{\overline{\mathrm{\Phi }}}_i\}$ and each metric yielded $|r|<0.25$, $|\rho |<0.20$, and $R^2<0.06$ for all linear fits. We then applied logit, arcsin transforms, and second-order polynomial regression, but the best $R^2$ remained below 0.10: this demonstrates that $\mathrm{\Phi }$ resides in $[-1,1]$ and classical metrics in $[0,1]$, yet no simple transform produces a strong linear relationship—hence our conclusion that the two capture distinct evaluation dimensions.

The observation that the malignant class exhibits more dispersed $\mathrm{\Phi }$ values follows from summary statistics on the pairwise entries. If we denote $P=\{(i,j):i<j\}$ as all network pairs, we compute for each class $c$:

$${\mu }_c=\frac{1}{|P|}\sum _{(i,j)\in P}\mathrm{\Phi }(N_i,N_j,D_c),{\sigma }_c=\sqrt{\frac{1}{|P|}\sum _{(i,j)\in P}(\mathrm{\Phi }(N_i,N_j,D_c)-{\mu }_c{)}^2}.$$

For the malignant melanoma class, we found ${\mu }_{\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{a}\mathrm{n}\mathrm{t}}\approx 0.30$, ${\sigma }_{\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{a}\mathrm{n}\mathrm{t}}\approx 0.18$, whereas for the benign class ${\mu }_{\mathrm{b}\mathrm{e}\mathrm{n}\mathrm{i}\mathrm{g}\mathrm{n}}\approx 0.10$, ${\sigma }_{\mathrm{b}\mathrm{e}\mathrm{n}\mathrm{i}\mathrm{g}\mathrm{n}}\approx 0.11$.

The larger standard deviation and the broader range of $\mathrm{\Phi }$ values in the malignant class reflect greater model disagreement in high-risk cases, underscoring potential clinical significance when consensus among classifiers is low.

Limitations

The EyePACS dataset is known to contain noisy labels due to inter-observer variability in grading. The ISIC dataset presents class imbalance, with fewer melanoma cases compared to benign lesions. While we did not apply additional strategies to address these issues, this choice aligns with the focus of our study, which was to evaluate the reliability indicator rather than to maximize predictive accuracy.

A further limitation of this study is that we restricted our analysis to CNN-based architectures. Although EfficientNet and Vision Transformers represent more recent models, their inclusion was beyond the scope of this work due to computational constraints (EfficientNet) and the need for different interpretability approaches (e.g., attention rollout for transformers) that are incompatible with our framework. Future work should extend the proposed reliability indicator to these architectures to evaluate its robustness across diverse model families.

Other main limitations of the current formulation lies in the definition of the similarity function $\mathrm{\Lambda }(a,b)$, which is constrained to the range $[0,1]$ by the application of a rectified version of SSIM. In particular, the current formulation:

(9) $$\mathrm{\Lambda }(a,b)=\omega (\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}(a,b))\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\omega (x)=\{\begin{array}{cc}0,\hfill & \mathrm{i}\mathrm{f}x<0\hfill \\ x,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}.$$

This truncation ensures interpretability of $\mathrm{\Lambda }(a,b)$ but may also reduce the sensitivity of $\mathrm{\Phi }$ to strong divergences, since all negative SSIM values are treated equally as zero, but it does not fully exploit the information provided by anti-correlated activations since negative SSIM values are nullified.

A refined formulation could preserve this information, potentially leading to a more nuanced characterization of divergence in feature selection. A sensitivity analysis, or the use of alternative similarity measures that preserve anti-correlation information, will be essential to assess the practical impact of this design choice.

Additionally, the current indicator penalizes disagreement but does not reward positive consensus. The definition of $\eta (N_x,N_y,I_i)$ could be generalized to account for premiality by introducing weighted constants P and L as follows:

(10) $$\eta (N_x,N_y,I_i)=\{\begin{array}{cc}P,\hfill & \mathrm{i}\mathrm{f}(C_y=C_x)=C_i\hfill \\ L,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}.$$

P represents a reward for correct joint classification and L a penalty otherwise. This formulation would allow $\mathrm{\Psi }(N_x,N_y,D)$ to reflect the level of agreement in predictions and their correctness.

Our current definition of $\eta$ is label-based and does not distinguish between low-confidence agreement (e.g., $p=0.51$) and high-confidence agreement (e.g., $p=1.00$). A straightforward extension is to weight $\eta$ by model confidence. In the generalized form

(11) $$\eta (N_x,N_y,I_i)=\{\begin{array}{cc}{P}_i,\hfill & \mathrm{i}\mathrm{f}(C_y=C_x)=C_i,\hfill \\ L_i,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e},\hfill \end{array}$$one can set

(12) $$P_i=\frac{1}{2}\left[p_{C_i}^{(N_x)}(I_i)+p_{C_i}^{(N_y)}(I_i)\right],L_i=-\frac{1}{2}\left[p_{C_x}^{(N_x)}(I_i)+p_{C_y}^{(N_y)}(I_i)\right],$$where $p^{(N_x)}(I_i)\in [0,1{]}^K$ and $p^{(N_y)}(I_i)\in [0,1{]}^K$ are the softmax probability vectors predicted by networks $N_x$ and $N_y$, respectively. As an alternative fully probabilistic variant that maps to $[-1,1]$ without hard thresholding, one may define

(13) $$\tilde{\eta }(N_x,N_y,I_i)=2min\{p_{C_i}^{(N_x)}(I_i),p_{C_i}^{(N_y)}(I_i)\}-1,$$which increases only when both networks assign high probability to the correct class.

Another important point concerns the robustness of the indicator. Since $\mathrm{\Phi }(TNN,D)$ is computed over a specific train-validation-test split, it reflects the trustworthiness of a model set under particular data partitioning. Future work should investigate the stability of $\mathrm{\Phi }$ across multiple splits to assess whether the indicator generalizes or is sensitive to data shuffling. This aspect is relevant to evaluating whether trustworthiness is an intrinsic model property or an artifact of the experimental design.

From a methodological standpoint, it is worth noting that a distance-like function ${\delta }^{\ast }(I_i,N_x,N_y)$ could be defined using an inverted similarity operator:

(14) $${\omega }^{\ast }(x)=\{\begin{array}{cc}0,\hfill & \mathrm{i}\mathrm{f}x<0\hfill \\ 1-x,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}.$$and thus

(15) $${\delta }^{\ast }(I_i,N_x,N_y)={\omega }^{\ast }(\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}(\mathrm{S}{\mathrm{M}}_{\mathrm{x}}(\mathrm{i}),\mathrm{S}{\mathrm{M}}_{\mathrm{y}}(\mathrm{i}))).$$

In this context, ${\delta }^{\ast }$ measures dissimilarity rather than similarity, and would equal zero if and only if the two networks are perfectly correlated. Investigating whether ${\delta }^{\ast }$ satisfies the triangle inequality and other properties typical of a metric could open up a path toward formalizing $\mathrm{\Phi }$ within a proper mathematical framework.

The formulation of $\mathrm{\Phi }$ assumes that agreement in internal representations is a proxy for trustworthiness. However, this assumption requires further experimental validation, possibly including human-in-the-loop settings or reinforcement schemes where $\mathrm{\Psi }(.)$ influences model updating based on expert feedback. Such integration would move toward a symbiotic AI paradigm in which interpretability and performance co-evolve, guided by algorithmic and human criteria.

An important limitation of the current work is the absence of probabilistic calibration analysis, specifically the lack of integration with metrics such as the Expected Calibration Error (ECE). While the $\mathrm{\Phi }$ indicator measures internal consistency among CNNs, it does not consider the alignment between predicted probabilities and empirical frequencies. A model can achieve high accuracy and even high $\mathrm{\Phi }$ values while remaining poorly calibrated, potentially overestimating or underestimating risk by large margins: this is particularly critical in clinical applications, where predicted probabilities often guide threshold-based interventions. Recent studies have also shown that models optimized for fairness may sacrifice calibration performance, revealing inherent trade-offs in multi-objective optimization. As a result, future developments of the trustworthiness framework should explore the combination of $\mathrm{\Phi }$ with probabilistic calibration metrics to ensure both interpretability and actionable reliability in AI-assisted diagnosis.

Conclusion and future work

In conclusion, the proposed indicator represents a first step toward quantifying internal trust in AI systems, especially in healthcare, where decisions must be accurate but also transparent and dependable. While the initial results are promising, much work remains to formalize and validate $\mathrm{\Phi }$ in diverse clinical contexts.

Future work will involve validating the proposed reliability indicator across a broader range of clinical domains and datasets, as well as exploring its integration with emerging architectures such as transformers and hybrid radiomics–deep learning models. Additionally, user studies with clinicians could provide valuable insights into the practical applicability and trustworthiness of the indicator in real-world healthcare settings. Also, future studies may incorporate noise-robust training methods for EyePACS and class-balancing techniques for ISIC to examine whether the reliability indicator maintains its validity when model performance is further optimized.

Requirements

MATLAB R2023b, Deep Learning Toolbox, and a GPU with at least 12GB VRAM are recommended. The pipeline is also compatible with only CPU devices. However, the running time will be higher.

Datasets

We use the ISIC skin lesion dataset (Rotemberg et al., 2021) and the EyePAC dataset (Cuadros & Bresnick, 2009). Both datasets are public. To download the correct ISIC images for reproducing our experiment, please refer to the file ISIC_Image_Labels.txt provided in the repository. The full link to download datasets are reported in the previous sections (see footnotes).

Preprocessing

Scripts for image enhancement (histogram equalization, contrast adjustment, hole erosion) are included: otsu_he.m for ISIC and rgb9cer_improve.m for EyePAC.

Model training

The training pipeline is implemented in Step1_Run_CNN_Trainings.mlx, supporting 19 CNN architectures trained from scratch. Please note that the pipeline is also capable of performing experimentation using fine-tuning and transfer learning, but for the scope of this contribution, these features must remain deactivated to reproduce this experimentation. The provided code is configured to reproduce this experimentation.

Usage instructions

These steps must be performed to ensure reproducibility. In each script, Set rng(0) and rand(0). The provided scripts are pre-configured with these parameters.

To switch between Melanoma and RD experimentation, you need to adjust the dataset path, number of output classes, and experiment name, as specified in the provided code and README. Md file explains how to perform this action. Please note that results (trained models, metrics, confusion matrices) are saved in a structured folder hierarchy.

Trustworthiness computation

Scripts Step2_Generate_Phi_By_Pairs.mlx and Step3_Generate_CNN_Pairs_Labels_Predictions.mlx compute Grad-CAM similarity and prediction consensus. Final $\mathrm{\Phi }$ values can be computed via pre-configured Excel sheets in the repository.