Optimizing statistical evaluation of multiclass classification in diagnostic radiology: a study of the two-parameter multidimensional nominal response model

Introduction

Item response theory (IRT) is a statistical framework for constructing, evaluating, and scoring tests and questions to measure test-takers’ capabilities (Hays, Morales & Reise, 2000; Cappelleri, Jason Lundy & Hays, 2014; Gelman et al., 2013). During testing, carefully curated items are presented to evaluate test-takers’ potential. Through IRT, responses provided by test-takers to these items are scrutinized to assess test-takers’ abilities and item complexities. Items used in IRT can be defined for medical diagnosis; for example, test-takers and items of IRT correspond to radiologists and cases in diagnostic radiology, respectively. Consequently, IRT can evaluate the outcomes of observer studies for medical diagnoses. Nishio et al. (2020) used IRT to evaluate radiologists’ ability to detect bone metastasis and brain infarction.

Generally, test-takers’ response data consist of 1 and 0 s for applying IRT, where 1 represents a correct response and 0 represents an incorrect one. However, binary responses do not suit all medical diagnostic scenarios, as multiple types of responses may be involved. For instance, radiologists’ responses in the N factor of cancer TNM grading may include four types (N0, N1, N2, and N3). Consequently, applying IRT to radiologists’ responses in multiclass classification is not straightforward.

The nominal response model (NRM), a multiclass extension of IRT, can be utilized for multiclass classification responses (Carpenter et al., 2017; Luo & Jiao, 2017). Despite reports on Bayesian NRM (Luo & Jiao, 2017), its stability is not guaranteed (The Stan Forums, 2020). Thus, multidimensional NRM (MDNRM) was proposed in a previous study (Nishio et al., 2023). While previous research successfully achieved model convergence using MDNRM (Nishio et al., 2023), its performance as a Bayesian model and statistical evaluation of estimated parameters were not assessed.

This study aimed to enhance the original MDNRM from previous research (Nishio et al., 2023). Hereafter, our improved version of the MDNRM is referred to as the two-parameter MDNRM (2PL-MDNRM). While the original MDNRM was developed from the one-parameter MDNRM (1PL-NRM) in the previous study (Nishio et al., 2023), the current study extended the conventional NRM (2PL-NRM) for 2PL-MDNRM. The development of 2PL-MDNRM was intended to make it possible to efficiently apply MDNRM to medical diagnosis (especially, diagnostic radiology). Generally, the ground truth of multiclass classification is used in the research of medical diagnosis or diagnostic radiology. Therefore, to improve the original MDNRM, the ground truth was considered in 2PL-MDNRM more rigorously than the original MDNRM. To validate the enhancement, values of widely applicable information criterion (Watanabe-Akaike information criterion, wAIC) (Watanabe, 2013, 2010) and Pareto-smoothed importance sampling leave-one-out cross-validation (LOO) (Vehtari et al., 2015) were calculated for the original MDNRM and 2PL-MDNRM. In addition, estimated parameters obtained using 2PL-MDNRM were statistically evaluated. The source code of this study was disclosed as open source through GitHub (https://github.com/jurader/2PL-MDNRM).

Materials and Methods

Dataset

A public dataset from a previous study (Nishio et al., 2022) was used in this study (accessible at https://github.com/jurader/MDNRM/blob/main/ground_truth_and_results.csv). Since a public dataset was used, institutional review board approval or informed consent was not required. The public dataset used in the present study contained classification results obtained from radiologists as described in two previous studies (Nishio et al., 2023, 2022). In addition, the ground truth were included for each item of the dataset. This dataset contained three classes of medical diagnoses: normal, non-novel-coronavirus pneumonia, and novel coronavirus pneumonia, resulting in a 3 × 3 confusion matrix. The dataset consists of 900 entries (150 cases × 6 radiologists), with each nominal response denoted as 0, 1, or 2, representing normal, non-novel coronavirus pneumonia, or novel coronavirus, respectively. Figure 1A shows the representative examples of the ground truth and radiologists’ response. Figure 1B shows the summary of a radiologist’s responses as confusion matrix.

2PL-MDNRM

The conventional NRM (2PL-NRM) was extended to 2PL-MDNRM. The ability parameter of test-taker ( $\theta )$ in 2PL-NRM becomes a multidimensional matrix in 2PL-MDNRM. In addition, the discrimination and easiness parameters ( $\alpha$ and $\beta$) can be multidimensional vectors. The following equations represent the three subtypes of the 2PL-MDNRM:

(4) $$\begin{array}{rl}& Pr\left(r_{ij}=t|groundtruthofitemi=s\right)={\displaystyle \frac{\exp \left(z_{ijst}\right)}{\sum _{h=1}^c\exp \left(z_{ijsh}\right)}}\\ & z_{ijst}={\alpha }_{is}{\theta }_{jst}+{\beta }_s\end{array}$$

(5) $$\begin{array}{rl}& Pr\left(r_{ij}=t|groundtruthofitemi=s\right)={\displaystyle \frac{\exp \left(z_{ijst}\right)}{\sum _{h=1}^c\exp \left(z_{ijsh}\right)}}\\ & z_{ijst}={\alpha }_s{\theta }_{jst}+{\beta }_{is}\end{array}$$

(6) $$\begin{array}{rl}& Pr\left(r_{ij}=t|groundtruthofitemi=s\right)={\displaystyle \frac{\exp \left(z_{ijst}\right)}{\sum _{h=1}^c\exp \left(z_{ijsh}\right)}}\\ & z_{ijst}={\alpha }_{is}{\theta }_{jst}+{\beta }_{is}.\end{array}$$

The three subtypes of 2PL-MDNLM shown in Eqs. 4–6 are referred to as 2PL-MDNRM_a, 2PL-MDNRM_b, and 2PL-MDNRM_r. In these subtypes, the discrimination parameter ( $\alpha$), easiness parameter ( $\beta$), and both discrimination and easiness parameters ( $\alpha$ and $\beta$) are multidimensional. The relationship between 1PL-NRM, 2PL-NRM, original MDNRM, and 2PL-MDNRM is shown in Fig. 2.

Results

From the Eqs. 3–6, the parameter dimensions of MDNRM subtypes are summarized as follows.

Original MDNRM

${\theta }_{jst}$: (number of test-takers) × (number of classes) × (number of classes)

${\beta }_{is}$: (number of items) × (number of classes)

2PL-MDNRM_a

${\theta }_{jst}$: (number of test-takers) × (number of classes) × (number of classes)

${\alpha }_{is}$: (number of items) × (number of classes)

${\beta }_s$: (number of classes)

2PL-MDNRM_b

${\theta }_{jst}$: (number of test-takers) × (number of classes) × (number of classes)

${\alpha }_s$: (number of classes)

${\beta }_{is}$: (number of items) × (number of classes)

2PL-MDNRM_r

${\theta }_{jst}$: (number of test-takers) × (number of classes) × (number of classes)

${\alpha }_{is}$: (number of items) × (number of classes)

${\beta }_{is}$: (number of items) × (number of classes)

The seven models of the original MDNRM and 2PL-MDNRM were applied to the public dataset to analyze the abilities of six radiologists across three classes. Parameters were estimated from the 900 nominal responses of the public dataset for each model.

All estimated parameters in each of the seven models had Rhat values of less than 1.10, indicating stable convergence as Bayesian models. In fact, the Rhat values are close to 1.00 in the optimal model, indicating good model convergence. This was a major contrast between the Bayesian NRM in the previous study (Luo & Jiao, 2017; The Stan Forums, 2020) and the seven models in the present study. Table 1 and Figs. 3 and 4 present the wAIC and LOO values for these models, with ranges of 988–1,176 and 1,121–1,207, respectively. Based on these values, the truncated normal distribution outperforms the gamma distribution for 2PL-MDNRM_a, 2PL-MDNRM_b, and 2PL-MDNRM_r. Notably, the smallest wAIC and LOO values (988 and 1,121, respectively) were observed in 2PL-MDNRM_r with the truncated normal distribution and 2PL-MDNRM_a with truncated normal distributions. These two models were identified the best-performing models in the present study.

Subsequent to this, the estimation results for the easiness and discrimination parameters are omitted as our focus is on the ability parameters. For the best-performing models identified (2PL-MDNRM_r with a truncated normal distribution in wAIC and 2PL-MDNRM_a with a truncated normal distribution in LOO), Tables 2 and 3 present the estimation results of the ability parameters for the six radiologists. Each radiologist’s ability is represented by a 3 × 3 matrix in the MDNRM. Higher diagonal values in the ability parameter matrix indicate higher test-taker’s abilities. Tables 2 and 3 include the estimation results for the 54 ability parameters (6 × 3 × 3).

As representative examples, the ability parameters for novel coronavirus pneumonia were compared using PD between radiologists 1 and 5 (between ${\theta }_{122}$ and ${\theta }_{522}$). Figures 5–7 depict the representative PD results for the three MDNRM models. ${\theta }_{122}$ was significantly better than ${\theta }_{522}$ for the two best-performing models: 2PL-MDNRM_r with a truncated normal distribution and 2PL-MDNRM_a with a truncated normal distribution. Conversely, ${\theta }_{122}$ was not significantly better than ${\theta }_{522}$ for the original MDNRM. Given that the wAIC and LOO values of the original MDNRM were not as favorable as those of the two best-performing models, statistical significance was not attained for the original MDNRM.

Discussion

Our study focused on addressing the stability of convergence and comparing the performance of different models in the context of multiclass classification of diagnostic radiology using the MDNRM. We found that all seven models demonstrated successful convergence, as indicated by Rhat values consistently below 1.10. Notably, the models of 2PL-MDNRM_r with the truncated normal distribution and 2PL-MDNRM_a with truncated normal distribution emerged as the top performers, achieving the smallest wAIC and LOO values, respectively. The PD results further confirmed the feasibility of statistically comparing ability parameters in these two best-performing models.

Given the previous instability of convergence in the conventional NRM (The Stan Forums, 2020), the significance of multidimensional ability parameters for stable convergence in MDNRM is suggested in the previous study (Nishio et al., 2023) and the current study. In addition, our results showed that the Bayesian models’ goodness of fit (wAIC and LOO values) was influenced by parameter dimensions and the use of a truncated normal distribution. The preference for the truncated normal distribution over the gamma distribution for the prior distribution of discrimination parameters is supported by previous studies (Röver et al., 2021; Gelman, 2006), hence the truncated normal distribution was adopted in the present study. Our results show that the parameter dimensions were important for improving the wAIC and LOO values, which means that the original MDNRM cannot adequately grasp the data generation process of multiclass classification.

In the original NDNRM and 2PL-NDNRM, the test-takers’ abilities were represented as a matrix. The diagonal values of this matrix are crucial in MDNRM; higher diagonal values indicate greater test-takers’ ability. Conversely, lower easiness parameter values indicate more challenging items. The discrimination parameters in MDNRM are similar to those in 2PL-IRT (Hays, Morales & Reise, 2000; Cappelleri, Jason Lundy & Hays, 2014). Therefore, the discrimination parameters can be utilized to judge whether the test items can be used to easily evaluate test-takers (radiologists) (Cappelleri, Jason Lundy & Hays, 2014; Nishio et al., 2023).

Despite achieving stable convergence in the original MDNRM, previous studies did not statistically compare ability parameters. Our results demonstrate the feasibility of such a comparison using PD and the optimal 2PL-MDNRM. This combination of PD and 2PL-MDNRM may offer a useful alternative to conventional statistical tests for evaluating estimated parameters.

While the default value of “94%” for PD in ArviZ packages was employed in this study, careful consideration must be given to its selection. Although “p = 0.05” is commonly used as the significance threshold in conventional statistical tests, it is not absolute, especially in cases of multiple testing where adjustments such as the Bonferroni or Holm corrections are necessary. Similarly, “94%” for PD must be carefully chosen based on the characteristics of studies. However, if the threshold of PD is carefully chosen, PD results in Bayesian models can be used instead of p-value in conventional statistical test.

Conventional models of latent response theory, such as item response theory or multidimensional nominal response model, have been used in clinical and psychometric studies (Zhao et al., 2023; Revuelta & Ximénez, 2017; Falk & Ju, 2020). These studies mainly focus on the response obtained from the experiment where the ground truth is not available or defined. On the other hand, the development of 2PL-MDNRM is intended to be used for the studies where the ground truth is available. This point is a major difference between conventional models and 2PL-MDNRM. Another major difference is the dimensionality of the ability parameters. Although D-dimensional parameters (D-dimensional vector) were used in the conventional multidimensional nominal response model, CxC-dimensional parameters (CxC-dimensional matrix) were used in 2PL-MDNRM. In addition, although the constrained versions of the multidimensional nominal response model were used in psychometric studies, the constraint was not necessary in 2PL-MDNRM.

This study had some limitations. First, although 900 nominal responses were utilized, the minimum number required for stable convergence remains unclear. Second, statistical significance depends on the models’ goodness or power, but this study did not elucidate how the goodness of fit influences the statistical significance of PD. Third, only one dataset was used in this study. As a result, it was not possible to evaluate whether the results of the current study could be reproduced in other datasets. Evaluating the generalizability of 2PL-MDNRM is a future study. Fourth, we did not evaluate multiple types of prior distributions. For example, use of the standard normal distribution was not evaluated in this study. Because the source code of this study is publicly available, we hope that the effect of the prior distribution will be evaluated rigorously in a future study. Fifth, multivariate normal distribution was not used as prior distribution for multidimensional latent traits. Sixth, although three types of 2PL-MDNRM were suggested in the current study, other variants of MDNRM can be suggested. For example, the ability parameter can be a multidimensional vector instead of a multidimensional matrix. We believe that the usefulness of other variants can be evaluated using the wAIC or LOO values.

Conclusions

The proposed 2PL-MDNRM offers a robust solution for multiclass classification in diagnostic radiology. Achieving successful convergence in parameter estimation akin to other Bayesian models, the superiority of 2PL-MDNRM over the original MDNRM is evident from the wAIC and LOO values. Our results also shows that the use of wAIC and LOO values may be useful for selecting Bayesian models. In addition, the feasibility of comparing ability parameters using PD underscores its potential utility in this context.