Quiz-style online training tool helps to learn birdsong identification and support citizen science

Overview of TORI-TORE

We developed a quiz-based birdsong training tool, TORI-TORE, to evaluate the effectiveness of automatically personalized adaptive training. Users can access the tool through a web browser and are identified by cookie at the specified URL (NIES, 2023). Users can access bird sound files stored on the server and choose bird species in a multiple choice quiz. TORI-TORE judges correctness, displays the results on the terminal, and stores correct and user-selected species in addition to quiz choices in the server database (Fig. 1).

TORI-TORE mainly consists of “test” and “quiz training” modules for evaluation and training, respectively. Quiz training consists of five choices (one correct answer and four distracters) for convenience. Only the correctly selected species will be judged as “correct”, and any other choice will be considered “incorrect”. In the quiz training module, after selecting a species name, the answer screen shows the correct species, the species selected by the user, the correctness, the sound source for each choice, a photo and spectrogram of the correct species, and if the species was “mastered” (described in “Adaptive training algorithm”). In the adaptive group, the history of correct answers affected the next quiz (described in “Adaptive training algorithm”). The goal is to acquire bird songs by repeated listening while taking the quiz training and checking answers. For the test module, see Study Design and Data Collection. There is no time limit for modules in TORI-TORE. TORI-TORE was coded in Perl v5.26.3 and is implemented as a CGI script with jQuery 3.4.1.

Adaptive training algorithm

We developed an adaptive training algorithm that tailors questions based on an individual’s proficiency level, one element of adaptive learning, for efficiently memorizing bird songs. The algorithm was designed (1) to reduce the frequency of bird songs (correct choices) once memorized and (2) to make incorrect alternatives easier when the user was not very proficient in the correct choice and harder when he or she was more proficient.

For creating the adaptive training algorithm, we referred to Tsumori & Kaijiri (2007). To help students memorize vocabulary, Tsumori & Kaijiri (2007) developed an algorithm able to automatically determine questions to be asked depending on the students’ understanding of the vocabulary, using multiple-choice questions in which the level of difficulty was controlled. The specific algorithms were as follows (Fig. 2). All species were given an initial proficiency level of zero. If the user selected the correct choice, the user’s the proficiency level of it increased by one. Selecting an incorrect choice reduced the user’s proficiency levels of the incorrect alternative and the correct choice by one. However, the proficiency level could not decrease below zero. The correct choice was basically selected from species with a proficiency level of less than three; accordingly, the probability of it being included as a correct choice in the training decreased when the proficiency level of the species was more than two (“mastered”). However, mastered species had a certain probability of being included in the training for review. The review probability P is formulated: $P=\frac{p\left(w{N}_u\right)}{N_u+w{N}_m}$, where p is max review probability (set at 0.25 in this study), w is review weight (set at 0.5), N_u is number of unmastered species, and N_m is number of mastered species. In the beginning of the training, the correct choices were randomly determined because proficiencies for all species were less than three.

As the proficiency for a bird species gradually changed, alternatives changed accordingly. In other words, the incorrect alternatives were determined by the proficiency level for the correct choices. When the proficiency level of the correct choice was low, species with high proficiency levels were selected as incorrect alternatives. As the proficiency level of the correct choice increased, species with low proficiency levels or “similar” species were selected as incorrect alternatives. “Similar” species were defined as species with similar songs to those of the correct choice, as determined by bird experts. In this study, there are zero to one similar species per correct answer choice (the algorithm allows more than one to exist). The degree of similarity was set to two levels (easier and more difficult level; see Table 1).

Case study

Study design and data collection

Participants were recruited in FY 2020. Eighty-four university students (from freshman to senior undergraduates) participated in this study. Participants were from the Kanto region, mainly Ibaraki Prefecture, and had no previously involvement in research or extracurricular activities related to birds. They gave consent for their participation and data use by reviewing a consent letter and checking a consent box.

Participants were assigned by stratified randomization to the adaptive training group (hereinafter, “adaptive group”) or the quiz training group, in which choices were selected at random (hereinafter, “baseline group”). Groups were matched with respect to academic year and gender, and were divided using random numbers.

Participants agreed in advance to the following: (1) they would be randomly assigned to two different quiz training groups, (2) they must not check information about birds on external sites other than the URLs presented in TORI-TORE, (3) they must not discuss the content of the experiment with others, (4) they would receive an honorarium only if they completed the entire experiment.

In the test module, to ensure that the test did not affect the training effect, correct answers were not disclosed to participants until after the delayed test. All 26 target species were included as choices to facilitate evaluation of changes in test scores. The experiment was conducted from January 12 to February 1, 2021. Participants were instructed to conduct the training on the schedule described in Table 2 and Fig. 3.

The number of valid responses (i.e., participants who completed the experiment) was 66, including 35 (53.0%) males and 31 (47.0%) females. Seventeen each were freshmen (18–20 years old), sophomores (19–21 years old), and seniors (21–23 years old) and 15 were juniors (20–22 years old). A total of 18 participants withdrew from the study.

Questionnaires

We developed web-based structured questionnaires to understand participants’ pre-training experience with nature and attitudes towards birds, to assess the impact of the training, and to improve the tool (Article S1 and Table S2). The questionnaires consisted of pre- and post-training questionnaires and questions were based on studies of the effectiveness of online training in citizen science projects and awareness among conservation activity participants (White, Eberstein & Scott, 2018; Starr et al., 2014; Ratnieks et al., 2016; Fukasawa et al., 2017a; Takase, Furuya & Sakuraba, 2014).

The pre-training questionnaire included 12 items across four sections. Section one was related to attitudes towards birds. On a 5-point Likert scale, participants ranked their interest in birds (1 = “Very interested” to 5 = “Not at all interested”), including birdwatching and learning bird songs. They were also asked to rank their birdwatching and birdsong identification experience level as “No experience”, “Beginner”, “Intermediate”, or “Advanced”. The second section focused on participants’ personal experience with nature, including experience with nature over the past year or environmental activities and pet ownership. The third section focused on motivations to participate in the survey. The final section (sociodemographic status) focused on the participants’ background, specifically their hometown.

The post-training questionnaire consisted of 22 items across two sections. Section one was related to attitudes towards birds. Using a 5-point Likert scale, participants ranked changes in their interests in birds (1 = “My interest in birds has really changed” to 5 = “My interest in birds has not changed at all”) and the level of interest in birds (1 = “Very interested” to 5 = “Not at all interested”), including birdwatching and learning bird songs. On a 5-point Likert scale, participants indicated whether they knew the species name in TORI-TORE and had ever heard the bird songs. The second section focused on the usage of TORI-TORE. We used a 5-point Likert scale to rank satisfaction (1 = “I’m satisfied with this training” to 5 = “I’m not satisfied with this training”) and evaluations (1 = “Strongly agree” to 5 = “Strongly disagree” and 1 = “There were too many questions” to 5 = “There were few questions”). We also used a 5-point Likert scale to assess whether participants perceived the adaptive training algorithm (1 = “Very much” to 5 = “Not at all”) (Article S1 and Table S2). Finally, we asked for feedback as an open-ended question.

Data analysis

To determine whether TORI-TORE was effective, we compared species identification test results and participants’ attitudes based on pre- and post-training questionnaires (see Code S1 and Data S1). Generalized linear mixed models (GLMMs) with a binomial error distribution were used to evaluate whether the training methods (adaptive training and baseline training) affect the posttest score considering the pretest score for each question. As a response variable, a dummy variable was set to 1 if the test was answered correctly and 0 otherwise. As explanatory variables, a dummy variable was set to 1 if the participant had taken the midterm test after adaptive training and 0 otherwise; 1 if the participant had taken the posttest after adaptive training and 0 otherwise; 1 if the participant had taken the delayed test after adaptive training and 0 otherwise; 1 if the participant had taken the midterm test after baseline training and 0 otherwise; 1 if the participant had taken the posttest after baseline training and 0 otherwise; 1 if the participant had taken the delayed test after baseline training and 0 otherwise. The dummy variables were fixed effects, while participants and tested species were included as random slopes and intercepts in the model. In addition, ordered logit models were used to evaluate whether the training method affects the post-training questionnaire responses about interests in birds considering pre-training responses. As a response variable, the questionnaire responses were set, and as explanatory variables, a dummy variable was set to 1 if the participant had taken adaptive training and 0 otherwise and to 1 if the participant had taken baseline training and 0 otherwise.

Second, a GLMM was used to understand whether the test scores after training differed between adaptive training and baseline training groups (see Code S1 and Data S1). The response variable was dummy variables set to 1 for correct answers in each test (midterm test, posttest, or delayed test) and 0 otherwise. The explanatory variable was a dummy variable taking a value of 1 for adaptive training and 0 for baseline training. The dummy variables were fixed effects, while participants and tested species were included as random slopes and intercepts in the model. The Wald tests were conducted to understand whether the posttest scores for each species after training differed between adaptive and baseline groups with the null hypothesis that the difference between the groups was zero. In addition, we used generalized linear models (GLMs) with binomial error distribution or ordered logit models to evaluate whether post-training questionnaire responses differed between groups. GLMs were used when there were two discrete choices, and the ordered logit model was used when there were more than two choices. The response variable was the answer to the questionnaire (five levels: see Article S1 and Table S2), and the explanatory variable was a dummy variable that was set to 1 for adaptive training and 0 for baseline training. To make it easy to interpret the ordered logit models, option numbering was adjusted so that the numbers assigned to positive responses were larger and those assigned to negative responses were smaller. For example, “very interested” was set to 5 and “not at all interested ” was set to 1.

We evaluated the hypothesis that a large number of quiz training questions, short lag times between quiz training and testing (hereinafter “lag times”), and long question intervals improve species identification test (posttest) scores (from the theories of the forgetting curve (Ebbinghaus, 1885)) (see Code S1 and Data S1). First, to understand whether adaptive training affected these factors, we performed MANOVA. We used a dummy variable taking a value of 1 for adaptive training and 0 for baseline training as an explanatory variable, the number of quiz training questions, inverse lag time (/days), and median question interval as the response variables. There was multicollinearity in the number of quiz training questions because the total number of questions in both groups was equal to 200. For this reason, we calculated p-values based on modified ANOVA-type statistics (Friedrich, Konietschke & Pauly, 2021), which can incorporate multicollinearity among response variables. Values were largest when tested immediately after training and were smaller but not equal to zero as time elapsed. We then ran a GLMM with these variables as the explanatory variables and a dummy variable as the response variable, taking a value of 1 for the correct answer in each test and 0 otherwise, to determine whether these variables affected the scores. The explanatory variables were fixed effects, while participants and tested species were included as random slopes and intercepts in the model.

The GLMMs and GLMs were implemented with the glmmTMB function in the glmmTMB package for R version 4.1.0 (Brooks et al., 2017; R Core Team, 2020). The ordered logit models were implemented with the clm function in the ordinal package for R version 4.1.0 (R Core Team, 2020; Christensen, 2019). The MANOVAs were computed using the manova function implemented in R version 4.1.0 (Brooks et al., 2017; R Core Team, 2020). In the case of multicollinearity, the MANOVA.wide function was used in the MANOVA.RM package for R version 4.1.0 (Friedrich, Konietschke & Pauly, 2021; R Core Team, 2020).

Ethics statement

Approval for this study was granted by the University of Tsukuba’s Faculty of Life and Environmental Sciences Ethics Committee (Subject number 2020-1). All tests, training, and questionnaire responses were anonymous. Each participant assigned a unique number as an identifier to match tests and training datas, and questionnaires for a given individual.

Informed consent was obtained from all participants. We explained the following to the participants before the experiment. Participation in the experiment was determined by the participants’ own free will. Therefore, they would not be disadvantaged in any way if they did not agree to take part in this experiment. In addition, even after consenting to participate in the experiment, they may withdraw from participation at any time and would not be disadvantaged by this.

Effects of TORI-TORE

In the midterm test, posttest, and delayed test, scores for both groups were significantly higher than those in the pretest (GLMMs: p-value for the Z-statistic < 0.001, Table S3). Scores for both groups increased from the pretest to midterm test and midterm test to posttest, but decreased from the posttest to delayed test (Fig. 4, GLMMs: p-value based on the Z-statistic < 0.001, Table S3). In particular, scores were 3.65 (±0.21) in the pretest, 11.45 (±0.57) in the midterm test after 2 days of training (100 questions), 14.62 (±0.71) in the posttest after 4 days of training (200 questions), and 12.45 (±0.69) in the delayed test.

Both groups showed increased interest in birds, bird watching, and learning bird songs based on pre- and post-training questionnaire responses (ordered logit models: p-value based on the Z-statistic < 0.01). For interest in birds, six participants in the adaptive group (22.2%) and six participants in the baseline group (15.4%) answered “very interested” or “interested” (positively) before the training, compared with 21 (77.8%) and 33 (84.6%) after the training. For interest in bird watching, 10 (37.0%) and nine (23.1%) responded positively before the training, compared with 21 (77.8%) and 33 (84.6%) after the training. For interest in birdsong learning, seven (25.9%) and 18 (46.2%) responded positively before the training, compared with 21 (77.8%) and 33 (84.6%) after the training.

Comparison between adaptive training and baseline training

In the midterm test, the adaptive group scored 9.4 (SE ±0.7) and the baseline group scored 12.8 (SE ±0.8). In the posttest, scores were 12.7 (SE ±1.2) and 15.9 (SE ±0.8). In the delayed test, scores were 10.9 (SE ±1.1) and 13.6 (SE ±0.9) respectively. Scores for the baseline group were significantly higher than those for the adaptive group in all tests (Fig. 4 and GLMMs: p-value based on the Z-statistic = 0.0021, 0.037, 0.073, respectively, Table S4). Baseline training had a more positive influence than that of adaptive training on test scores in the comparison between the pretest and the midterm test with a wider score gap. However, adaptive training had a more positive influence than that of baseline training from the midterm test to posttest and had a less negative impact than baseline training on the change from the posttest to delayed test, with a narrower difference in scores between the two groups (Fig. 4 and GLMMs: p-value based on the Z-statistic < 0.001, Table S4).

In the posttest, there was a variation in the accuracy rate for each species and each group (Fig. 5). There were significant differences in the scores between the groups (Wald test summary in Table S5) for Common Pheasant (Phco, p = 0.033), Grey Wagtail (Moci, p = 0.00040), Japanese Pygmy Woodpecker (Deki, p = 0.0048), Oriental Greenfinch (Chsi, p = 0.030). In the adaptive training algorithm, similar species were included as choices when the proficiency level for a species increased; accordingly, the adaptive group was expected to have a higher accuracy rate for similar species. Contrary to this expectation, the accuracy rate for similar species was not higher in the adaptive group compared with the baseline group, except for large-billed crow (Coma) and carrion crow (Coco).

There was no difference between groups in questions related to attitudes towards birds on the post-training questionnaire (ordered logit models: p-value based on the Z-statistic > 0.05). Regarding the “usage of TORI-TORE”, responses in both groups were generally positive. Regarding whether participants perceived the adaptive training algorithm, the adaptive group exhibited significantly different responses to the following items: “Wrong species in the quiz were followed by more questions in the quiz”, “The frequency of questions on the mastered species decreased”, “As the proficiency level of the species of the correct choice increases, a similar species is selected as the incorrect alternative (but not at lower proficiency levels)”, “While the species of the correct choice was less proficient, the more proficient species was selected for the incorrect alternatives, and as the proficiency level increased, the less proficient species were selected for these incorrect alternatives” (ordered logit models: p-value based on the Z-statistic < 0.05). In other words, participants of adaptive group recognized that they were receiving adaptive training. Adaptive training had a significantly negative effect on the responses about satisfaction (ordered logit models: estimate = −1.307, p-value based on the Z-statistic = 0.0121). No significant differences between groups were found for other questions (ordered logit models: p-value based on the Z-statistic > 0.05).

Factors that contribute to training effectiveness

Variables affected by training methods

As expected, there were significant differences in the effects of the number of quiz training questions, the inverse lag times, and the median question intervals for each of the 26 species at the time of the posttest between the adaptive and baseline groups (MANOVA: p < 0.001, p = 0.0094, p = 0.066, respectively, Table S6).

Variables affecting test scores

The number of quiz training questions, inverse lag time, and median question interval (explanatory variables), irrespective of group, influenced test scores (response variable) (GLMM: p- value based on the Z-statistic < 0.001, respectively, Table S7). Specifically, a large number of quiz training questions, a long inverse lag time (few lag times), and a long question interval had a positive effect on test scores. In other words, the training method affected these three parameters, which in turn affected test scores.

Relationship between explanatory variables and the accuracy rate for each species

For the relationship between number of quiz training questions and the accuracy rate for each species, the number of quiz training questions tended to be 6–9 for the baseline group and 4–10 for the adaptive group (Fig. 6). In the adaptive group, a higher number of quiz training questions corresponded to species with lower accuracy rates. In addition, for the random slope for the effect of each species (i.e., how much the score improves for each quiz training question) and the number of quiz training questions, the correlation coefficients were −0.223 (p = 0.27) for the baseline group and −0.546 (p = 0.0039) for the adaptive group. For the relationship between inverse lag time and the accuracy rate for each species, the baseline group was concentrated at 0.8–1.0 /days (1–1.25 days), while the adaptive group was concentrated in 0.6–1.1 /days (0.91–1.67 days) (Fig. 7). In the adaptive group, an increased inverse lag time (i.e., to the right of the graph) corresponded to species with lower accuracy rates in the adaptive group. In addition, for the random slope for each species (i.e., how well the response improves with the size of the inverse lag time) and the inverse lag time, the correlation coefficient was −0.0172 (p = 0.93) for the baseline group and −0.237 (p = 0.24) for the adaptive group. For the relationship between median question interval and the accuracy rate for each species, the distribution differed from those for two explanatory variables above (Fig. 8). Compared with the baseline group, the adaptive group tended to have more species with narrower intervals (e.g., the adaptive group had 18 species with values below 20, compared with 13 species for the baseline group). A higher median question interval (i.e., to the right of the graph) corresponded to species with higher accuracy rates and a lower median question interval corresponded to species with the lower accuracy rates in the adaptive group (more than in the baseline group). In addition, for the random slope for each species (i.e., how well the response improves with the size of the median question intervals) and the median question interval, the correlation coefficient for the adaptive group was 0.320 (p = 0.11) and for the baseline group was 0.475 (p = 0.014).

For Common Pheasant (Phco), Grey Wagtail (Moci), Japanese Pygmy Woodpecker (Deki), and Oriental Greenfinch (Chsi), where there were significant differences in scores between groups on the posttest, the results in terms of accuracy rate, tendency to answer incorrectly, and each explanatory variable are as follows. Regarding the difficulty level, the mean accuracy is medium for Grey Wagtail (Moci), Japanese Pygmy Woodpecker (Deki), and Oriental Greenfinch (Chsi), but the accuracy of Common Pheasant (Phco) was exceptionally high (Fig. 5). In addition, there is not much difference in the number of quiz training questions for Grey Wagtail (Moci), Japanese Pygmy Woodpecker (Deki), and Oriental Greenfinch (Chsi) between groups, but the number of quiz training questions for the adaptive group of Common Pheasant (Phco) is lower than that for the baseline group (Fig. 6). For Oriental Greenfinch (Chsi), the adaptive group tended to answer more times and at shorter question intervals. By the adaptive group, Oriental Greenfinch (Chsi) tended to be misidentified to species with larger number of quiz training questions and narrower intervals. Grey Wagtail (Moci) tended to be misidentified as other wagtails (Japanese Wagtail (Mogr) and White Wagtail (Moal)) by the adaptive group, and Japanese Wagtail (Mogr) and White Wagtail (Moal) were more likely to be listed as a similar species in the same choice in the adaptive training (Table 1, “Adaptive training algorithm”). However, the song of Grey Wagtail (Moci) is not similar to that of Japanese Wagtail (Mogr) and White Wagtail (Moal), and was not listed as a similar species. In addition, the adaptive group had narrower question intervals than the baseline group for Oriental Greenfinch (Chsi) and Grey Wagtail (Moci) (Fig. 8). For Japanese Pygmy Woodpecker (Deki), however, no trend was observed in the number of questions submitted and the intervals by themselves, or the species misidentified.