In silico study of medical decision-making for rare diseases: heterogeneity of decision-makers in a population improves overall benefit

Context

In this paper, we study the two-armed-bandits problem in clinical settings in which there are two new treatments for a disease without knowledge of success, denoted A and B. Each patient’s Bernoulli outcome, favourable (success) or unfavourable (failure), is recorded. The true rates of favourable outcomes for A and B are unknown, and they are denoted a and b, with 0 < a, b < 1. A series of patients select A or B, one by one, and the next (n + 1)^th patient is informed with the preceding outcomes that n patients have been treated in total and n_A and n_B have selected A and B, respectively, with n_As and n_Bs successful outcomes and n_Af and N_Bf failures, respectively. As shown in Table 1, n = n_A + n_B, n_A = n_As + n_Af, n_B = n_Bs + n_Bf, n_s = n_As + n_Bs, and n_f = n_Af + n_Bf.

Beta conjugate distribution to binomial outcomes

The outcome of each treatment is a Bernoulli outcome, and the parameter of unknown success rate follows a binomial distribution. In Bayes’ theorem, whereby the posterior is proportional to the prior multiplied by likelihood, there is one advantage in that the beta distribution is the conjugate distribution to Binomial outcomes (shown as Note S1). When no patient has been treated, we set a uniform Beta(1,1) as the initial prior, and then the two parameters of the prior are to be updated by the outcomes, successes n_∗s + 1 and failures n_∗f + 1. (1)${\displaystyle p_{\mathrm{\ast }}\left(\theta |n_{\ast s},n_{\ast f}\right)=\beta \left(n_{\ast s}+1,n_{\ast f}+1\right)=\frac{1}{\mathrm{B}\left(n_{\mathrm{\ast }s}+1,n_{\mathrm{\ast }f}+1\right)}{\theta }^{n_{\mathrm{\ast }s}}{\left(1-\theta \right)}^{n_{\mathrm{\ast }f}},}$

where ∗indicates A or B, β and B indicate the beta distribution and function, respectively, and 0 ≤ θ ≤ 1.

Optimistic/pessimistic individuals in a population with heterogeneity of decision-making

We assumed that every individual selects one out of two treatments (A or B) with a higher value that is estimated from the beta posterior distribution with given current outcomes (successes and failures) of each treatment, p_∗(θ|n_∗s, n_∗f) (Eq. 1). In this study, we modelled two types of individuals; one type of individual selects a treatment based on the posterior mean v_∗E (Eq. 2). They select the treatment with the larger posterior mean/larger maximum expected success rate (Eq. 5). We call this type of individuals’ selection as the E-strategy, where E stands for “Expected”. The other type of individual is somehow optimistic or pessimistic and selects the treatment with a higher value that is different from the expected value and depends on each individual’s optimistic/pessimistic preference. We set an attitude index, w, to parameterize the two preferences of individuals, where the w of pessimistic individuals ranges from −1 to 0 and the optimistic w ranges from 0 to 1. In fact, in terms of treatment assignment in clinical settings, we assumed that optimistic individuals care whether the treatments are adequately successful or not and that they set a target value (t) higher than the maximum posterior mean (v_∗E) (Eq. 4), and calculate the probability that success rate is higher than the t. This probability is denoted by $v_{\ast T}\left(t\right)$ (Eq. 3). By contrast, we assumed that pessimistic individuals set tlower than the maximum posterior mean (v_∗E), and calculate the probability that success rate is higher than t. The modelled t is calculated from a function of maximum posterior mean (v_maxE) but depending on this individual’s attitude index w (Eq. 4). Those individuals who are optimistic or pessimistic select the treatment with higher v_∗T (Eq. 5). We refer to this type of individual’s decision as a T-strategy, where T stands for “Target”.

The posterior mean of the success rate is (2)${\displaystyle v_{\mathrm{\ast }E}={\int }_0^1\theta \times p_{\mathrm{\ast }}\left(\theta |n_{\ast s},n_{\ast f}\right)d\theta =\frac{n_{\mathrm{\ast }s}+1}{n_{\mathrm{\ast }s}+n_{\mathrm{\ast }f}+2},}$ where ∗ indicates A or B

The probability of a success rate more than a target value is (3)${\displaystyle v_{\ast T}\left(t\right)={\int }_t^1{\mathrm{p}}_{\ast }\left(\theta |n_{\ast s},n_{\ast f}\right)d\theta ,}$ where ∗ indicates A or B, and the target value t is calculated by v_maxE = max(v_AE, v_BE) and depends on the attitude index w. (4)${\displaystyle t=\left\{\begin{array}{c}\mathrm{w}+\left(1-\mathrm{w}\right)v_{maxE},0\le \mathrm{w}\le 1\hfill \\ \left(1+\mathrm{w}\right)v_{maxE},-1\le \mathrm{w}<0\hfill \end{array}\right.}$

where positive and negative w stands for optimism and pessimism, respectively. Correspondingly, the v_maxE ≤ t ≤ w specifies optimism, and 0 ≤ t ≤ v_maxE specifies the pessimism.

Subsequently, the probability of selecting A, Prob(A), is given as (5)${\displaystyle Prob\left(A\right)=\left\{\begin{array}{c}1:v_{A\mathrm{\ast }}>v_{B\mathrm{\ast }}\hfill \\ 0.5:v_{A\mathrm{\ast }}=v_{B\mathrm{\ast }}\hfill \\ 0:v_{A\mathrm{\ast }}<v_{B\mathrm{\ast }}\hfill \end{array}\right.,}$ where ∗ indicates E or T. Actually the selection of every individual is deterministic based on the values that are calculated from the 2 by 2 table values in principle. The selection is stochastic only when the values are equal. When we assumed a population was homogeneous, their selection was deterministic except for the stochastic selection due to the identical values for two arms. When we assumed a population was heterogeneous, the individuals’ optimistic/pessimistic attitude vary among them and the sequence of individuals were stochastically generated in the experiment.

We assumed that the population is a mixture of individuals with various levels of optimism/pessimism. To specify the heterogeneous population in this model, we assumed that need w is symmetric around zero and is in a monomodal distribution ranging from -1 to 1. With this assumption, the majority of people are almost neutral and relatively few people are strongly optimistic or pessimistic. As a simple model for this distribution, we assumed (6)${\displaystyle w\sim 2\left(\beta \left(u,u\right)-0.5\right),}$ where u parameterizes the shape of distribution of w.

An example for the selection was given in the Fig. 1, where the Beta posterior distributions of two treatments were drawn with the information: the eighteen (18 = (14 − 1) + (6 − 1)) patients have been treated with the treatment A, resulting in outcomes of 13 successes and five failures, and three (3 = (3 − 2) + (2 − 1)) patients have been treated with B, resulting in outcomes of two successes and one failure. Two sets of decision values based on E-strategy (v_∗E) and T-strategy (v_∗T) were calculated separately, and in the Fig. 1 v_∗E are indicated as vertical lines (v_AE = 0.7 and v_BE = 0.6), and v_∗T are indicated as the area under the curve truncated by a vertical line ($v_{AT}\left(t\right)=0.16$ and $v_{BT}\left(t\right)=0.18$). Since 0.7 > 0.6, individuals with E-strategy should select A. Since 0.16 < 0.18, individuals with T-strategy (t = 0.8 corresponds to an optimistic individual with w approximately 0.3) should select B.

Experimental conditions

The true but unknown success rate of the two arms, a and b, and the number of total patients, N, are parameterized. The total 5,050 pairs of combination of a and b was generated with 0 < b ≤ a < 1. For the T-strategy population, w as an attitude index is parameterized within the range of −1 and 1. In addition, we evaluated relatively small N values, from 1 to 100, because our objective is to study the effects of optimism/pessimism of decision-makers for treatment assignment with a rarer disease in a clinical setting where the patient population size is small. Under the same condition of a, b, and N, we first compared the overall success rate between the homogeneous population who take the E-strategy and the homogeneous population who take the T-strategy, where all T-strategy individuals have the same w values.

After the comparison between the homogeneous E-strategy population and the homogeneous T-strategy population, we investigated the effects of heterogeneity. We generated a population with the T-strategy whose w values were not same but distributed as shown in Eq. (6), where u was 5, 30, 70 and 500.

Measure of the overall benefit

Typical case of homogeneous E-strategy (E.st) and homogeneous T-strategy (T.st)

Individuals in the homogeneous E.st population select treatments based on the posterior mean, v_∗E,. Individuals in the homogeneous T.st population select treatments based on v_∗T, which represents the optimism/pessimism attitude and is shared by all the individuals in the population. Figure 2 is the results of the experiment, where the success rate of the two treatments were a = 0.8 and b = 0.6,  andthe optimistic attitude w of the T.st population was 0.5. The total number of patients was N = 100. Figures 2A and 2B are the 2-dimensional histograms, where one axis is the fraction of individuals who selected A and the other axis is the overall success rate when all the processes reached N = 100 for E.st and T.st, respectively. The processes of selection by a series of individuals are stochastic; the fraction of selecting A and the overall success rate take distribution. The exact distribution of the fraction and the rate were calculated and displayed. In the case of E.st, as shown in Fig. 2A, the distribution was bimodal, with the higher peak corresponding to the occasions in which a majority of N patients had selected the better treatment arm, A, and the lower peak indicating that the minority had selected the inferior treatment arm, B, with a subsequently lower OSR. In the case of T.st, as shown in Fig. 2B, the distribution was monomodal with the peak towards the A-arm selection. The bimodality was the result of the exploitation-exploration dilemma. In some cases, the patients who selected the inferior one turn out to be successful with the expected success rate higher than the true success rate because this is a stochastic process. In this case, the following patients tend to select the inferior treatment arm with the belief that this treatment arm has a high success rate, and they lose the chance to select the other treatment arm that was truly better. Panels A and B indicate that the decision strategy of the population affected the exploitation-exploration pattern.

Figures 2C and 2D show the average fraction of individuals who selected the A-arm and the OSRs among a number of patients from 1to 100 of the two strategies, respectively. Panel C shows that the A-arm fraction of E.st and T.st at N = 100 was 0.714 and 0.855, respectively, and the fraction was higher for T.st throughout for the number of patients. Panel D shows that OSR of E.st and T.st at N = 100 was 0.743 and 0.771, respectively, and the rate was higher for T.st throughout. This finding was also related to the lack of exploration that occurred in E.st in this particular scenario.

Comparison of overall benefit of homogeneous decision-makers between E-strategy (E.st) and T-strategy (T.st) with the same attitude index w value

The typical case above showed that the homogeneous decision-makers of T.st with w = 0.5 outperformed E.st when a = 0.8 and b = 0.6 for N = 1, 2, …, 100. However, such superiority is not always true for all conditions of a, b, N,  and w. In fact, under some conditions, homogeneous T.st decision-makers outperformed the homogeneous E.st, but under other conditions, the E.st decision-makers outperformed the T.st.

We evaluated the difference of Overall Success Rates (OSRs) between homogeneous T.st and homogeneous E.st decision-makers with the same attitudes of w values for various conditions; N = 1, 2, …, 100 and $\left(a,b\right)=\left\{\left(a,b\right)|a,b\in \left\{0.01,0.02,\dots ,0.99\right\},a\geqq b\right\},w=${−0.0005, −0.001, …, 0.999, 0.9995}. The number of (a, b) pairs was 5,050.

Figure 3A visualizes the difference of OSR of 5,050 (a, b) pairs at N = 10, 30,  and 100 for w =  − 0.8,  − 0.4, 0.4, and 0.8. The (a, b) pairs form a triangular space. The horizontal axis is a, and the vertical axis is b. Red indicates (a, b) pairs where T.st outperforms E.st, and blue indicates (a, b) pairs where E.st outperforms T.st. Colour intensity stands for the value of the difference of OSRs as indicated in the colour bar on the right. The black curves stand for the (a, b) pairs without difference between E.st and T.st.

The colour patterns of 12 conditions of the Fig. 3A show that the superiority of two strategies is the function of (a, b) conditional to N and w. The homogeneous optimistic decision-makers (T.st with w > 0) outperformed E.st when both a and b were relatively large, but their performance was worse than E.st when both a and b were relatively small, as shown in the first and second rows of w = 0.4 and 0.8 of Fig. 3A. When N increases, the colour intensity tends to become stronger. In contrast, the performance of the homogeneously pessimistic attitude decision-makers (T.st with w < 0) was worse than E.stfor the majority of (a, b) pairs, and T.st outperformed only when both a and b were small. When N increases, the colour intensity tends to become stronger.

Next, we investigated the relation between (a, b) pairs and the superiority of E.st and T.st, regardless of the intensity of optimism or regardless of w values as far as w > 0. We calculated OSRs for w = {0, …, 0.999, 0.9995}. Fig. 3B coloured the (a, b) pairs with red, grey and blue, where red indicates that the average OSR of T.st is higher than the OSR of E.st for all optimistic w values, and blue indicates that the average OSR of T.st is lower than the OSR of E.st for all optimistic w values, and grey indicates otherwise. In general, T.st tends to outperform when both treatments have a relatively high success rate and E.st outperforms when both treatments have a relatively low success rate. In this comparison, we evaluated homogeneous T.st populations from which all individuals in a population were the same w values, and we set different such same w (ranging from −1 to 1) values for each of homogenous populations. In the next experiment, we modelled populations that are heterogeneous for decision attitudes that consisted of individuals whose optimism/pessimism attitude index w varied and compared their performance with E.st.

The effect of heterogeneity of decision-makers in a population

In reality, populations seem to consist of individuals with various attitudes. We modelled heterogeneity of optimism/pessimism attitude index w with Eq. (6), where w distributes symmetric around 0. Figure 4 is the colour plot to display the superiority of E.st andheterogeneous T.st. Four panels show the result of four different distributions of optimism/pessimism index w, specified with u = 5, 30, 70 and 500. The distribution of w is displayed in a window of each panel.

Figure 4 show some tendencies. The heterogeneous T.st tended to outperform E.st (red) when both a and b were relatively high. With smaller u or larger variance of w, the difference of OSR was bigger. Actually, the right most panel indicates essentially no difference between E.st and heterogeneous T.st when the variance of w is very small. When u is small (u=5), the triangular area was divided into two coloured subareas almost evenly, and when u is larger (u = 30 and 70), the area where the heterogeneous T.st was superior was bigger than the area where E.stwas better.