Using demographics toward efficient data classification in citizen science: a Bayesian approach

Data collection

The data used in this study were collected within a citizen science project for obtaining information about the status of the environmental health of the Gowanus Canal (Brooklyn, NY, USA) (Torre et al., 2019). The images were taken by an aquatic robot designed as part of the Brooklyn Atlantis project (Laut et al., 2014), which, over the years, was used to address a number of important questions in citizen science, from the effect of design interventions to face-to-face interactions with scientists and on to improving engagement in rehabilitation exercises (Laut et al., 2015, 2017; Nov, Laut & Porfiri, 2015; Cappa et al., 2016, 2018; Palermo et al., 2017a, 2017b; Diner et al., 2018; Nakayama et al., 2018; Torre et al., 2019).

Volunteers were asked to inspect the images of the Canal and identify the presence of objects that could endanger the environment (Torre et al., 2019). The images taken by the robot were uploaded on a temporary website built for this experiment, where volunteers could access them from their computers and mobile devices. Before taking part in the project, participants had to log in through either a Facebook profile or an email account to prevent them from performing the task more than once. After accessing the website, participants were first presented with a short movie about the scope of the project. Then, participants initiated a practice session, in which they were instructed to classify whether the object in the image would represent a potential threat to the environment by clicking either a “threat,” “no threat,” or “I don’t know” button below the image. After the task was performed, the correct answer was shown together with a brief explanation. Before the experiment, Torre et al. (2019) identified the correct answer of each image through careful examination and discussion, and the selection of images only included those which received a unanimous classification.

After the classification of two objects in the practice session, the main task started, and participants were asked to classify 31 images consecutively, which appeared on the screen for 5 s each. Participant could choose between “threat,” “no threat,” or “I don’t know” buttons, but this time, the correct answer was not displayed. If the participant did not select any answer in 5 s, it was recorded as “no answer.” To avoid possible confounding effects on performance, the order of the images’ display was randomized for each participant. Upon completing the classification task, the participants were asked to fill out a short questionnaire where they provided information on their education level and degree of interest toward the environment.

The data collection was carried out between February and June 2017, with a total of 91 volunteers recruited in the project. Here, we focus on the 88 of them who filled out the preliminary demographic questionnaire. All the participants were over 18 years old and their responses were anonymized. The data collection was approved by the institutional review board of New York University (IRB-FY2016-184).

Bayesian inference

Let us assume that a pool $\mathcal{V}=\{1,\dots ,n\}$ of volunteers participates in the binary classification of a set $\mathcal{I}=\{\mathrm{1,}\dots \mathrm{,}m\}$ of images. In the process of classification of image $i\in \mathcal{I}$, the unobservable binary parameter that we wish to estimate is denoted as θ_i. In our experiment, θ_i is equal to 1 if image i contains a threat for the environment, and it is equal to 2 otherwise. A priori, we assume that we have no cues on the possible content of that image, and therefore we set $$P_0\left({\mathrm{\theta }}_i=1\right)=P_0\left({\mathrm{\theta }}_i=2\right)=0.5$$ for all i, where the subscript 0 indicates that we refer to the probability at step 0, that is, before starting the classification process. After every successive classification, we propose to sequentially update these probabilities by using Bayes’ rule (Gelman et al., 2013). At each classification step, say j ≥ 1, the observable data is the classification y_il of image i performed by participant l = l(j), randomly selected from the pool $\mathcal{V}$ at step j. The possible outcomes of the observed variable y_il are 0, corresponding to a late reply (the participant does not classify within 5 s), 1 or 2, corresponding to the participant classifying the image as containing or not containing a threat, respectively, and 3, corresponding to an uncertain participant choosing the “I don’t know” option.

In a Bayesian framework, the behavior of the l-th participant is characterized by the conditional probabilities (1) $$P(y_{il}=\mathrm{\alpha }|{\mathrm{\theta }}_i=\mathrm{\beta })\mathrm{,}$$ for all α ∈ {0,1,2,3}, β ∈ {1,2}, and $i\in \mathcal{I}$. Since we do not know a priori whether some images are more difficult to classify than others, we assume that the probabilities in (1) are independent of i, and therefore, for all $i\in \mathcal{I}$, we write (2) $$P(y_{il}=\mathrm{\alpha }|{\mathrm{\theta }}_i=\mathrm{\beta })=P(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=\mathrm{\beta })\mathrm{,}$$ which represents the probability that the classification output of the l-th participant is equal to α, given that the image contains (or does not contain) a threat (depending on the value of β).

In this work, we propose that the behavior of a volunteer, say the l-th, is related to his/her demographics (such as motivations and educational level), encoded by a vector x_l of one or more integer variables. More specifically, we assume that the probabilities (1) depend on the variables x_l, which are therefore called explanatory in the Bayesian literature (Carlin, Louis & Carlin, 2000; Gelman et al., 2013; Garriga, Piera & Bartumeus, 2017). Accordingly, based on the classification performed by the participant l(j) randomly selected at step j, and on his/her demographics, the probability that image i contains a threat for the environment can be updated in a Bayesian fashion as follows: (3) $$\begin{array}{ll}{P}_j({\mathrm{\theta }}_i=1)\hfill & =\frac{P(y_l=\mathrm{\alpha },{\mathrm{\theta }}_i=1,x_l)}{P(y_l=\mathrm{\alpha },x_l)}=\frac{P(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=1,x_l),P(x_l,{\mathrm{\theta }}_i=1)}{P(y_l=\mathrm{\alpha }|x_l)P(x_l)}\hfill \\ \hfill & =\frac{P(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=1,x_l)P(x_l|{\mathrm{\theta }}_i=1)P({\mathrm{\theta }}_i=1)}{P(y_l=\mathrm{\alpha }|x_l)P(x_l)},\hfill \end{array}$$ for all j ≥ 1, where P_j(θ_i = 1) is defined as P_j(θ_i = 1|y_l = α, x_l), and we omit the explicit dependence of l on j to simplify the notation. Observing that x_l and θ_i are independent, we have P(x_l|θ_i = 1) = P(x_l), thus yielding (4) $$P_j({\mathrm{\theta }}_i=1)=\frac{P(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=\mathrm{1,}{x}_l)P({\mathrm{\theta }}_i=1)}{P(y_l=\mathrm{\alpha }|x_l)}\mathrm{.}$$

From the law of total probability, we can write (5) $$P(y_l=\mathrm{\alpha }|x_l)=P(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=1,x_l)P({\mathrm{\theta }}_i=1|x_l)+P(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=2,x_l)P({\mathrm{\theta }}_i=2|x_l).$$

Noting again the independence between x_l and θ_i, and substituting (5) into (4), we finally establish (6) $$P_j({\mathrm{\theta }}_i=1)=\frac{P(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=\mathrm{1,}{x}_l)P_{j-1}({\mathrm{\theta }}_i=1)}{P(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=\mathrm{1,}{x}_l)P_{j-1}({\mathrm{\theta }}_i=1)+P(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=\mathrm{2,}{x}_l)P_{j-1}({\mathrm{\theta }}_i=2)}\mathrm{,}$$ where we used as prior P_{j − 1}(θ_i = 1).¹ Once the conditional probabilities P(y_l = α|θ_i = β, x_l), for all α, β, and x_j, have been estimated on a sample of volunteers, then, as a new volunteer v decides to participate in the study, we only need access to the demographics x_v to characterize his/her behavior.

Setting a threshold 0.5 ≤ σ <1, we label the image as classified at the first step t ≥ 1 such that either P_t (θ_i = 1) > σ or P_t (θ_i = 1) < 1 − σ, and the final classification is (7) $${\widehat{\mathrm{\theta }}}_i=\arg \underset{\mathrm{\beta }\in \{1,2\}}{\max }{P}_t({\mathrm{\theta }}_i=\mathrm{\beta }).$$

The threshold σ can be viewed as the selected confidence level for the classification. Clearly, the higher σ is, the higher the accuracy would be, but this would require a larger number of volunteers to classify the image.

The effectiveness of the Bayesian inference is intrinsically related to our knowledge of the conditional probabilities in Eq. (6). If these probabilities were fully known, the more explanatory variables we considered, the faster P_j(θ_i = 1) would converge to either 0 or 1, thereby leading to a more efficient classification for a given confidence level σ. However, in real applications we can only perform sample estimations of these conditional probabilities, which are typically evaluated on a small dataset. Therefore, their accuracy might be undermined by the sample size, but also by a biased demographic distribution of the sample. Hence, a trade-off arises in the choice of the explanatory variables: adding variables increases the theoretical classification accuracy, but the sample estimation might become less accurate due to the reduced size of the sample on which the conditional probabilities are estimated. Therefore, in designing a Bayesian classification algorithm, a crucial point is the selection of how many and which explanatory variables should be considered.

Classification algorithm

We consider the degree of interest toward the environment and the level of education of the volunteers as possible explanatory variables. The interest toward the environment is encoded by the integer x_l1, ranging from 1 (participant l is “not at all” interested) to 5 (participant l is “very much” interested), while the education level is encoded by a second integer parameter x_l2, which increases from 1 (“high school diploma or less”) to 4 (“graduate or professional degree”) as the participant education level increases, while it is set to 5 if he/she prefers not to answer. Accordingly, this yields three possible choices for x_l: the behavior of the participant can be evaluated based only on the degree of interest toward the environment (x_l = x_l1), on the education level (x_l = x_l2), or on both explanatory variables (x_l = [x_l1 x_l2]^T, where the superscript T means matrix transposition).

For any possible choice of x_l, adopting a Bayesian approach for classification requires a preliminary estimation of the participants’ accuracy based on their demographics. Specifically, this consists in estimating the conditional probabilities (8) $$P(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=\mathrm{\beta }\mathrm{,}{x}_l)\mathrm{,}$$ for all α ∈ {0, 1, 2, 3}, β ∈ {1, 2}, and all possible values of x_l. To this aim, we consider the set of volunteers $\mathcal{V}$ who filled out the demographic questionnaire, and partition it in two groups, denoted $\mathcal{T}$ and $\mathcal{C}$, respectively. The set $\mathcal{T}$ encompasses the volunteers used to compute the sample estimations (9) $$\widehat{P}(y_l=\mathrm{\alpha }|{\mathrm{\theta }}_i=\mathrm{\beta }\mathrm{,}{x}_l)$$ of the conditional probabilities in Eq. (8), and is called training set in the following, while the set $\mathcal{C}=\mathcal{V}-\mathcal{T}$ is used for testing the performance of the Bayesian approach. Namely, each image $i\in \mathcal{I}$ is classified as a result of the following steps:

• Initialization: the prior is set to ${\widehat{P}}_0({\mathrm{\theta }}_i=\mathrm{\beta })=P_0({\mathrm{\theta }}_i=\mathrm{\beta })=0.5$, β = 1, 2, and the set of volunteers available for classification at step 0 is ${\mathcal{A}}_0=\mathcal{C}$; a threshold σ is selected in the interval $\left(0.5,1\right)$;

• Step j ≥ 1: a participant l = l(j) is randomly selected in ${\mathcal{A}}_{j-1}$, which is updated as (10) $${\mathcal{A}}_j={\mathcal{A}}_{j-1}-\{l(j)\}\mathrm{,}$$ and the estimated probabilities ${\widehat{P}}_j(\mathrm{\theta }=\mathrm{\beta })$, β ∈ {1,2}, leveraging the sample estimations (9), are computed as (11) $${\widehat{P}}_j({\mathrm{\theta }}_i=1)=\frac{\widehat{P}(y_l={\mathrm{\alpha }}_{il}|{\mathrm{\theta }}_i=1,x_l){\widehat{P}}_{j-1}({\mathrm{\theta }}_i=1)}{\widehat{P}(y_l={\mathrm{\alpha }}_{il}|{\mathrm{\theta }}_i=1,x_l){\widehat{P}}_{j-1}({\mathrm{\theta }}_i=1)+\widehat{P}(y_l={\mathrm{\alpha }}_{il}|{\mathrm{\theta }}_i=2,x_l){\widehat{P}}_{j-1}({\mathrm{\theta }}_i=2)},$$ and ${\widehat{P}}_j({\mathrm{\theta }}_i=2)=1-{\widehat{P}}_j({\mathrm{\theta }}_i=1)$, where α_il is the output of the classification of image i performed by participant l; and

• Termination: the algorithm terminates at the first step t such that either ${\mathcal{A}}_t=\varnothing$ or (12) $$\max \left\{{\widehat{P}}_t({\mathrm{\theta }}_i=1)\mathrm{,}{\widehat{P}}_t({\mathrm{\theta }}_i=2)\right\}>\mathrm{\sigma }\mathrm{.}$$

  Similar to Eq. (7), the i-th image is classified as $${\widehat{\mathrm{\theta }}}_i=\arg \underset{\mathrm{\beta }\in \{1,2\}}{\max }{\widehat{P}}_t({\mathrm{\theta }}_i=\mathrm{\beta }),$$ and the number of participants used to classify image i is recorded as n_i = t.

Performance analysis

Out of the 91 volunteers who participated in the study, we focus on the 88 who filled out the questionnaire, so that $\left|\mathcal{V}\right|=88$. Our goal is to determine whether the Bayesian approach can successfully leverage demographic information, and which individual attributes should be used as a proxy of reliability. Furthermore, we seek to evaluate the impact on the overall performance of the termination threshold σ, which is varied in the set (0.5,1) with step 0.02. Then, for each value of σ and for all the three possible selections of x_l, we evaluate the performance of the classification algorithm in terms of the average number ν of volunteers employed, computed as $\nu ={\displaystyle {\sum }_{i\in \mathcal{I}}}{n}_i/\left|\mathcal{I}\right|,$ with n_i being the number of volunteers considered to classify the i-th image, and of classification accuracy χ, evaluated as the fraction of the 31 images that is correctly classified. The ground truth used to evaluate χ is represented by the preliminary classification performed by Torre et al. (2019).

We notice that the performance of the classification algorithm might be biased by the choice of the set of volunteers $\mathcal{T}$ employed for estimating the conditional probabilities (8), and by the specific classification order of the volunteers in the set $\mathcal{C}$. We set the cardinality of the set $\mathcal{T}$ to 45, which is approximately half of the total number of volunteers, and, to avoid potential biases, we randomly pick m = 10,000 alternative selections ${\mathcal{T}}_i$, i = 1,…,m, of the set $\mathcal{T}$, and for each i we consider p = 100 random permutations of ${\mathcal{C}}_i=\mathcal{V}-{\mathcal{T}}_i$. Then, for each possible choice of σ and x_l, we compute the mean values $\overline{\chi }$ and $\overline{\nu }$ as (13) $$\overline{\chi }(\mathrm{\sigma }\mathrm{,}{x}_l)=\frac{1}{mp}{\displaystyle \sum _{i=1}^m}{\displaystyle \sum _{j=1}^p}{\chi }_{ij}(\mathrm{\sigma }\mathrm{,}{x}_l)\mathrm{,}\overline{\nu }(\mathrm{\sigma }\mathrm{,}{x}_l)=\frac{1}{mp}{\displaystyle \sum _{i=1}^m}{\displaystyle \sum _{j=1}^p}{\nu }_{ij}(\mathrm{\sigma }\mathrm{,}{x}_l)\mathrm{,}$$ where χ_ij and ν_ij are the accuracy and average number of volunteers employed when using ${\mathcal{T}}_i$ as the training set and considering the j-th permutation of ${\mathcal{C}}_i$ as the classification sequence, respectively.

For comparison purposes, we use the majority voting approach (Kestler et al., 2011) as a reference. Namely, we consider the outcome of the classification when using the same sequence and number of participants used for Bayesian estimation, and compute its average value ${\overline{\chi }}_{mv}(\mathrm{\sigma }\mathrm{,}{x}_l)$ for all σ and for all the three possible choices of x_l. A complementary metric is the percentage π(σ, x_l) of all trials where the accuracy of the Bayesian approach overcomes that of majority voting.

To further delve into the performance difference between the two approaches and clarify the impact of the threshold σ, we present receiver operating characteristic (ROC) curves, typically employed to compare and select binary classifiers (Fawcett, 2006). For each value of the threshold σ, the ROC curve depicts the true positive rate (TPR) against the false positive rate (FPR). The TPR is defined as the fraction of real positives (the image contains a threat) that are correctly classified as positive, while the FPR is the fraction of real negative (the image does not contains a threat) that are incorrectly classified as positive. Then, for each value of the threshold σ, we extract a scalar unbiased measure of accuracy, the area under the curve (AUC) (Powers, 2011). We remark that, as the threshold σ modulates the number of participants employed to classify an image, and not the rate of positives, the ROC curves might not be monotone as in standard ROC analysis (Fawcett, 2006).

Preliminary analysis of the citizen science data

In total, 88 volunteers filled out the demographic questionnaire. Table 1 presents the demographic composition of the pool of volunteers, while Tables 2 and 3 describe the distribution of the classifications outputs depending on the degree of interest toward the environment and on the education level, respectively. The χ² test for independence revealed that the distributions of answers were different among x_l1 ( ${\chi }_{20}^2$ = 100,320, p < 0.001) and among x_l2 ( ${\chi }_{10}^2$ = 25,813, p = 0.004). From visual inspection, we cannot identify any trivial relationship between the classification output and demographics. Lack of correlation is also supported by the Kendall rank correlation coefficients ρ₁ and ρ₂ between the fraction of images correctly classified and the variables x_l1 and x_l2, respectively. Although one might expect volunteers’ accuracy to be positively correlated both with their interest toward the environment and education, we found ρ₁ = − 0.06 and ρ₂ = − 0.02, suggesting an absence of a linear dependence.

A closer look at the conditional distributions can help identify some non-trivial relationships between the classification outputs and demographics. For instance, from Table 2 we observe that the number of late replies is the highest when participants are “very much” interested in the environment. This could suggest that the participants are afraid to misjudge the image and then click the wrong button, due to their genuine concern for the environment. At the same time, their percentage of false positives is the lowest, so they seldom generate a false alarm. Our Bayesian estimation algorithm has the potential of leveraging this kind of less trivial nonlinear relationships between volunteers’ accuracy and demographics.

Bayesian inference against majority voting

In Bayesian estimation, the selection of the most appropriate explanatory variables is crucial for boosting its performance. Although in principle the more explanatory variables we include, the better estimation we attain, the finiteness of the training sample requires a more thoughtful approach. In Fig. 1, we compare the performance for the three alternative choices of x_l, that is, the explanatory variables are either both the degree of interest toward the environment and education level (x_l = [x_l1 x_l2]^T), or just one of the two attributes (x_l = x_l1 or x_l = x_l2). From panel A, we see that, for all values of the threshold σ, the accuracy decreases when both explanatory variables are considered. This outcome can be explained by considering that the sample is too small ( $\left|\mathcal{T}\right|$ = 45) to allow for an accurate estimation of the conditional probabilities in Eq. (1) for all the 15 possible combinations of x_l1 and x_l2. Furthermore, we observe that the best performance is obtained when the interest toward the environment is used as the explanatory variable. This can be expounded by looking at the demographic composition of the pool. Indeed, from Table 1 we observe a more uniform distribution of the pool with respect to x_l1, while the level of education is skewed toward x_l2 = 4, as more than the 81% of the participants has a graduate or professional degree. This clearly limits the accuracy in the estimation of the conditional probabilities in (1) when x_l2 ≠ 4, thus explaining the superior accuracy associated to the choice x_l = x_l1.

The effectiveness of a Bayesian approach is also confirmed by a direct comparison with the majority voting. As one can note from Fig. 1A, for all possible choices of the explanatory variables x_l and the threshold σ, the average accuracy of the Bayesian algorithm is superior to majority voting when the same sequence of labelers is used. Furthermore, in all cases the percentage π(σ, x_l) of trials in which the Bayesian classification outperforms majority voting is larger than 72.7%, see Fig. 1B. Choosing x_l = x_l1 results in a higher performance, with a peak of π(σ, x_l) = 92.6% when σ = 0.98.

Figure 2 illustrates how the threshold σ can be used to modulate the tradeoff between accuracy and average number of volunteers employed. If the conditional probabilities (1) were known, both the accuracy and number of volunteers should monotonically increase with σ. However, this becomes nontrival when those probabilities are estimated, whereby a correct choice of the explanatory variable is crucial. For x_l = x_l2 and low values of σ the average accuracy $\overline{\chi }$ decreases with σ, but, when the optimal choice x_l = x_l1 is made, monotonicity is regained and the more participants we use, the more the accuracy improves.

These considerations are further confirmed by the ROC analyses in Fig. 3, as an alternative accuracy measure, the AUC, is also non-monotone with σ for x_l = x_l2, while for x_l = x_l1 we can tune σ to regulate the tradeoff between the AUC and $\overline{\nu }$. Moreover, the ROC curves highlight differences between the two classifiers, where we observe a shift of the curves toward the left, such that our Bayesian classifier strongly reduces the FPR. This comes at the price of a moderate decrease of the TPR.