The effect of using games in teaching conservation

Participants and course details

Our study was conducted on 55 conservation biologists from two courses: (1) a Wildlife Conservation Course (WCC) organised by Wildlife Conservation Research Unit (WildCRU, University of Oxford) and Nottingham University in Malaysia and; (2) the Recanati-Kaplan Centre Postgraduate Diploma in International Wildlife Conservation Practice in Oxford, United Kingdom (Table 1). Personnel were early career conservation biologists who have been working in conservation organisations for fewer than five years or were doing their graduate studies in conservation biology.

Twenty-one conservation biologists from different countries in Southeast Asia were selected to attend the WCC that took place between 19th and 30th January 2015. In 2016, the same course was held again on 4th to 15th January for 16 other participants. This course was held in the University of Nottingham Malaysia Campus (UNMC). Overall, participants consisted of 14 (37.8%) men and 23 (61.2%) women. One participant was 16–20 years-old, 16 participants were 21–25 years-old, 12 were 26–30 years-old, four were 31–35 years-old and four were >35 years-old. The course comprised topics ranging from concepts in conservation biology, to transferrable and practical skills. This study was approved by the University of Nottingham Malaysia Campus Science & Engineering Research Ethics Committee (Ref No. JL051115 & CT011114) and participants involved willingly signed an agreement form prior to the lessons.

Another set of participants comprised 16 international students (eight each for year 2015 and 2016) attending the seven-month Recanati-Kaplan Centre Postgraduate Diploma in International Wildlife Conservation Practice directed by WildCRU, University of Oxford. Four participants were 21–25 years-old, 10 were 26–30 years-old, one was 31–35 years-old and one was >35 years-old. There were four men and four women each year. The taught Diploma covered a range of subjects in conservation biology. In three to four topics of the Diploma, the participants were subjected to different lesson types and data on these participants collected for this study. In year 2016, two PhD students from the WildCRU attended the lesson for two topics and participated in the games and DI classes. They were included in the study to add to the sample size. This study was approved by the University of Oxford Central University Research Ethics Committee (Ref No: R42538) (Oxford University, 2017) and participants involved willingly signed an agreement form prior to the lessons. Table 1 summarises the cohort demographics for each course for separate years.

Personality and learning style questionnaires

Prior to the start of the course, questionnaires were administered to participants to assess their personality traits and learning styles. The Five Personality Factor test, 5PFT″ (70 statements) is the first published personality questionnaire specifically created to measure the personality traits now known as the big five (Elshout & Akkerman, 1975): agreeableness, conscientiousness, extroversion, neuroticism and openness. The big five model is one of the most established and recognised approaches in measuring individual differences in personality (Costa & McCrae, 1992).

A recent review identified 71 different learning style models (Coffield et al., 2004). It can thus be difficult researchers to select an appropriate learning style assessment (Cassidy, 2004). We used three criteria to select our learning style model: (1) whether the purpose of the model matches the aim of the study, (2) how extensively the model has been used in past studies and (3) the validity and reliability of the associated instrument (De Bello, 1990). Thus, we used the inventory of learning styles (ILS) (120 statements) (Vermunt, 1994). This model measured different aspects of processing strategies, regulation strategies, learning orientations and mental models of learning (Vermunt, 1994). Vermunt’s model of learning has been used widely within the higher education community (Busato et al., 1998a; Boyle, Duffy & Dunleavy, 2003) and has been used to examine the relationship between learning style, the big five personality traits and academic performance (Quinn et al., 2017; Busato et al., 1998b). Students answered the inventory and scores were subsequently calculated for each component of Vermunt’s four learning styles: undirected, reproduction-directed, application-directed and meaning-directed.

For both questionnaires, statements within each component overlapped considerably and were a mix of positive and negative, increasing the validity and reliability of the assessment (Mount, Barrick & Strauss, 1994; Vermunt, 1998). Details on the definition of each category of personality and learning style are provided in Method S1.

Lesson types

We employed three lesson types: DI, SG) or EG for our classes. The guidelines used to create the games are explained in Method S1. Each topic was divided into two sessions that had complementary content. For example, the first session on behavioural ecology was about how to conduct behavioural ecological research and second session was on the link between behavioural ecology and conservation biology. Table 2 provides information on the content taught for each session of each topic. For one topic, either one (SG or EG) or two games (SG and EG) were developed. This allowed for comparisons of the students’ responses to SG vs DI, EG vs DI or SG vs EG. For the Diploma 2016 students, the R analysis topic had three sessions each with a different lesson type (SG or EG or DI) to examine the response to three lesson types within one topic.

All topics were taught by the same tutor, irrespective of lesson type. This minimised any potential confounds when using different tutors. Regardless of the lesson type, the content and concepts taught for each session were the same. To ensure this, we used the same lesson slides for all lesson types, only adding slides to introduce the game in SG and EG. In all lesson types, students were grouped in teams of two to four people to discuss questions asked by the tutor and to work on the given tasks. This standardises the number of opportunities to answer questions in all lesson types.

Experimental design

We adopted two experimental designs to examine the differences between lesson types: repeated-measures design or counterbalanced measures design. In a repeated-measures design, a single group of students was taught the first session with one lesson type (DI, SG or EG) and then the second session with another lesson type (Fig. 1A) (Von Ende, 2001; Lawal, 2014). In a counterbalanced measures design, the class was divided into two groups and both were taught a two-session lesson. During the first session, one group was taught using one lesson type and the other was taught using the other lesson type. In the second session, the treatments were swapped and thus each group experienced both types of lesson in a balanced ordered manner (Fig. 1B) (Shuttleworth, 2009).

The design used was dependent on the number of students and the time available. The games required a minimum of eight players (for four teams with at least two persons each). Therefore, the repeated-measures design was used for the Diploma course as there were only eight students (Tables 1 and 2). We recognise that the repeated-measures design has a limitation in which students’ response to the second session depended on the lesson type of the first session, i.e. order effect (Von Ende, 2001; Lawal, 2014). Therefore, when possible, for the same cohort of students, we alternated the order of the lesson type for different topics to minimise potential treatment order effects (Table 2). Because of the limitations in time and resources (one tutor), not all students were exposed to all topics. However, we were able to ensure that all students except for the two PhD students were exposed at least twice to EG, twice to DI and once to SG to allow for within-individual comparisons. The two PhD students were exposed twice to EG and once to DI lesson types.

The counterbalanced measures approach utilised twice the amount of time and half the number of students each time as compared to the repeated-measures design. Therefore, the counterbalanced measures design would be limited to the course with more time and with a larger group of students, i.e. the WCC. This design reduces the chances of the treatment order influencing the results (Shuttleworth, 2009). For example, the behaviours of the subjects might be affected by the order in which treatments were given, due to fatigue or other reasons. However, due to time limitations, we also used the repeated-measures design for a few topics of the WCC (one topic in year 2015, two topics in 2016; Table 2). Details of the experimental design and order utilised for the different groups of students are provided in Table 2.

Assessment

Participants were assessed in three ways: (1) perception: via questionnaires administered to participants, collecting data of individual perceptions of each lesson (Supplemental Information 1); (2) behaviour: via observational surveys and video recording during the classroom sessions and; (3) learning: via quizzes pre and post-session to examine the knowledge acquired by each individual. These modes of assessment overlapped considerably and hence served to cross-validate the students’ responses to the different lesson types (Dirks, Wenderoth & Withers, 2014).

• (1) Perception

The questionnaire designed to test students’ perceptions was divided into three sections. In the first section, students were asked to compare the lesson to that of a traditional teaching method experienced previously (not during the course) and rate each statement (e.g. amount of content taught) according to whether they strongly disagreed (1 point), disagreed (2 points), no difference (3 points), agreed (4 points) and strongly agreed (5 points) (Supplemental Information 1). These statements reflected the wide range of perceptions held by the learners about their behaviour and learning (Black & Wiliam, 1998). In 2014, this first section of the questionnaire was tested on 20 undergraduates of the University of Oxford as well as on eight students of the Diploma course. Subsequently, it was edited by three of our co-authors prior to the conduct of this study (year 2015 onwards). This section was modified slightly after the earlier courses in 2015. The statement ‘Degree of connection with peers’ was added to the questionnaire after the first course (WCC 2015) and the statement ‘Appreciation of application’ was added for students of 2016. Sections 2 and 3 of the questionnaire were only administered to students of 2016. These extra statements and sections were added after considering the feedback from students in earlier courses in 2015: Students wrote in their feedback that these were the other beneficial aspects of game lessons. The sample sizes for these additional statements of the questionnaire are thus smaller than that of the other statements.

The second section of the questionnaire is a shortened version of the intrinsic motivation inventory (Ryan, Koestner & Deci, 1991), which focuses on the degree of intrinsic motivation and its predictors: perceived choice, perceived competence, pressure/tension and social bonding (see Method S1 for details). These four subscales each had five statements which overlapped considerably and were a mix of positive and negative, increasing the validity of the assessment (McAuley, Duncan & Tammen, 1989; Tsigilis & Theodosiou, 2003). The third section was aimed at measuring interpersonal interactions between students within teams, which is a positive predictor of intrinsic motivation (Ryan, Koestner & Deci, 1991). It was based on the Inclusion of Other in Self Scale (Aron, Aron & Smollan, 1992): ‘Please circle the picture that best describes your relationship to the other participants in your group’ (Supplemental Information File (Questionnaire)).

• (2) Behaviour

We noted the frequency of joyful behaviour (smiling, laughing, clapping, cheering), distracted behaviour (yawning, falling asleep expressions, looking at phones), asking questions and answering questions for each individual (Supplemental Information File (behavioural observation sheet)). We only recorded asked questions that were relevant to the topic, rather than any questions relating to the game or administration. Behaviours were selected based on three criteria. First, at similar classes prior to the study, a catalogue of the typical students’ behaviour was created and refined to answer our research questions (Supplemental Information File (Behavioural observation sheet)). Second, these behaviours complemented the statements used in the perception questionnaire, for example, the ‘degree of encouragement to ask questions’ and hence served to verify the perception response. Third, several of these behaviours have been used in previous studies on undergraduates to understand student participation (asking and answering questions, Fritschner, 2000; Mustapha, Rahman & Yunus, 2010) and distraction (phone-using and sleeping, McCoy, 2016; Mustapha, Rahman & Yunus, 2010). Particular behaviours were biased to certain lesson types, e.g. joyful behaviour is expected to be higher when engaging in a game than when attending a DI lesson. Nevertheless, we were also interested in examining whether these behaviours were different between the two game forms. For example, when a SG is distracting, it might elicit fewer joyful behaviours than a EG lesson. Due to logistical constraints, we did not record behaviour for the R analysis lessons in the WCC 2015. Behaviours that occurred for a duration of more than a minute were considered as multiple counts if there was at least a 10 s interval between the behaviours (Mahar et al., 2006). To standardise the number of opportunities of asking question, during the DI lessons, questions used in the game lessons were asked verbally and the tutor waited for answers from the participants.

Behavioural data were collected via observer note-taking and subsequently verified with video records (Mustapha, Rahman & Yunus, 2010). Observers were trained prior to lessons to minimise inter- and intra-observer bias. They were trained by JWL at lessons not analysed in this study. Two observers and one observer were employed for observations during the WCC and Diploma programme, respectively. Since courses were carried out in different countries, different observers were involved. Therefore, after the courses, another observer who was not involved in direct note-taking made behavioural recordings from the videos filmed during class. Our inter-observer correlation coefficients were 0.71, 0.78, 0.72 and 0.80 for Diploma 2015, Diploma 2016, WCC 2015 and WCC 2016, respectively and thus above the recommended criterion of >70% or greater (Jonsson & Svingby, 2007). To ensure consistency, we used the behavioural data collected by the last observer for our analyses.

• (3) Learning

Quizzes consisted of questions asking facts, meaning or application of topic to a real-world situation. Therefore, a question could be categorised as reproduction-, meaning- or application-based question type, guided by the revised Bloom’s taxonomy (Krathwohl, 2002) and complementary to the learning styles quantified by Vermunt’s ILS. The Cognition Process Dimension of the revised Bloom’s taxonomy consist of six levels that are defined as Remember, Understand, Apply, Analyze, Evaluate and Create. We only constructed questions that tested the lowest three levels (Remember, Understand, Apply) as the higher three levels required more complex testing procedures. Our questions on reproduction (remembering) required students to ‘retrieve, recognise and recall relevant knowledge from memory’ (Krathwohl, 2002); for example, ‘what is inbreeding depression? Our meaning-based questions required learners to ‘construct meaning from messages through interpreting, exemplifying, classifying, summarising, inferring, comparing and explaining’ (Krathwohl, 2002). An example is ‘What is the difference between raster and vector? Finally, our application-based questions required students to ‘carry out or use a procedure in a given situation’ (Krathwohl, 2002). An example of a question is ‘How do we apply the stratified random sampling method when interviewing villagers? The questions were in a multiple-choice format with four options, only one of which was right. There were 10 questions for each quiz, comprising a mixture of question types. The same questions were administered before and after each session. In 2016, a third quiz with the same questions was conducted one week after the lesson to quantify retention of knowledge after a longer period of time (as compared to immediately post-lesson). We determined the difficulty level and discriminatory level of each quiz by calculating the mean and standard deviation of correctness. If a question was more difficult, the mean ‘correctness’ would be closer to zero and if it were easier, the mean ‘correctness’ would be closer to one. The larger the standard deviation of correctness, the larger the variation in correctness among students and therefore the more discriminatory the question was.

Statistical analysis

All analyses were conducted using R 3.0.2 and with mixed models due to the repeated measures obtained (Ellison, 2017; Tango, 2017). To examine the effects of lesson type (DI, SG or EG) on students’ perceptions, we used a multinomial mixed model with level of agreement as the response, lesson type as the fixed factor, and student and topic as random factors. Student was entered as a random factor as there were multiple questionnaires completed by each student (e.g. different sessions but same topic). Topic was entered as a random factor to account for any potential differences among topics. The covariates included were year, course (WCC or Diploma), age, gender, teaching experience (yes or no), formal teaching training (yes or no). We used one model for each statement of the perception questionnaire (e.g. Encouragement to ask questions).

We analysed the variation in each behaviour (joyful, distracted, question-asking and question-answering) with two analyses: (i) a generalised linear mixed model (GLMM) with a binomial error distribution, the ‘probability of performing the behaviour’ as a response variable, and (ii) a linear mixed model with a normal error distribution and the ‘rate of behaviour per hour’ as the response variable, omitting zeros.

To examine whether learning effectiveness differed with lesson type for the three topics, we entered in a GLMM, with binomial error distribution, correctness as a response (1 = correct; 0 = wrong), lesson type and period (before or after or one-week post lesson (for 2016 only)) and their interaction as fixed factors, topic and student as random variables. Details on checking for assumption violations, and fixed factors and random factors entered for each analysis are provided in Method S1.

To account for variation in question difficulty and discriminatory level, we also conducted GLMM analyses with the standardised ‘correctness’ as the response. That is, each individual response (0 or 1) was standardised to the mean ‘correctness’ obtained by all individuals answering that question and then we divide this difference by the corresponding standard deviation of the ‘correctness.’ Therefore, if a question were more difficult, the mean ‘correctness’ would be closer to zero and if it were easier, the mean ‘correctness’ would be closer to one.

Perception

Overall, students perceived EG as the most beneficial and SG as intermediately beneficial among the lesson types. Students scored EG and SG significantly higher than DI in the following questionnaire statements: ‘remembrance of content,’ ‘motivation to learn after lesson,’ ‘broadens perspective,’ ‘nurtures creativity,’ ‘challenging,’ ‘attention retention,’ ‘engagement with tutor’ and ‘engagement with student’ (Figs. 2A–2C; Tables S1 and S2). For the statement ‘learning from peers’ only EG was scored significantly higher than DI, SG attained an intermediate score between EG and DI (Figs. 2A–2C; Table S1). EG was scored significantly higher than SG and DI in the questionnaire statements ‘connection with peers’ and ‘appreciation of application’ (Figs. 2A and 2C). SG was scored significantly higher than DI in the questionnaire statement ‘better understanding.’ Lastly, for statements ‘amount of content’ and ‘encouragement to ask questions,’ no significant differences were found among lesson types (Figs. 2A and 2C; Tables S1 and S2).

Intrinsic motivation measured only for 2016 students revealed significantly higher scores of ‘interest/enjoyment’ for EG than for DI (SG attained an intermediate score; Fig. 2D). This self-report measure of intrinsic motivation was not explained by ‘perceived competence,’ ‘perceived choice’ nor ‘pressure or tension’ which are predictors of intrinsic motivation (Fig. 2D). Bondedness rating with peers within the team was significantly higher for EG than for DI (Fig. 2E).

When examining whether the overall average rating was affected by personality and learning styles, all except ‘Undirected learning style’ had non-significant interactions with lesson type on overall rating (Undirected learning score*lesson type: χ²₁ = 7.29, p = 0.026; Table S1). The correlation between overall rating and ‘Undirected learning’ scores was negligible for EG, and positive for DI and SG (Fig. S1).

Behaviour

Similar to the results on perception, students’ overall behavioural records were most positive in the EG among the three lesson types. There was a significant difference in the probability of ‘asking question,’ ‘joyful behaviour’ and ‘distraction’ occurring among lesson types. Students displayed a higher probability of question-asking for EG lessons compared to DI and SG lessons, higher probability of joyful behaviour in EG and SG lessons than in DI lessons and higher probability of distracted behaviour in DI than in SG and EG lessons (Fig. 3A; Tables S3 and S4). The frequency of ‘joyful’ behaviour per unit time was significantly higher in EG compared to SG and DI lessons and the frequency of ‘distracted’ behaviour per unit time was significantly lower in EG compared to DI lessons (Fig. 3B; Tables S3 and S4).

Learning

Questions varied in their mean correctness and thus difficulty level (Table S5). They also varied in their standard deviation of correctness and hence level of discrimination (Table S5). On average, questions were sufficiently difficult (mean correctness across lessons ranged from 0.35 to 0.55) and were able to discriminate different degrees of learning (standard deviation of correctness across lessons ranged from 0.144 to 0.244) (Table S5).

Overall, there was no significant difference in students’ learning among the lesson types. Students scored higher in quizzes after the lesson than before the lesson. One-week post lesson quiz scores varied (only 2016 students), depending on lesson type. The average one-week post lesson scores were similar to that of the after-lesson quiz.

When analysing all quiz questions, there was no significant difference in learning effectiveness among lesson types (DI vs SG vs EG), as shown by the lack of interaction between lesson type and period (before or after) on the proportion of quiz answers being correct (χ²₁ = 5.18, p = 0.075; Fig. 4A). This lack of significance of lesson type was consistent across both years (Figs. 4B and 4C; Table S6). However, when examining only reproduction-type questions, there was a significant interaction between lesson type and period on the proportion of quiz answers being correct (χ²₁ = 30.0, p < 0.001; Table S6): the proportional increase in quiz score was higher for DI and EG than for SG (Fig. S2A). This effect was largely attributed to the quiz performance of 2015 students (Figs. S2Ai and S2Aii). When analysing meaning-directed and application-directed question scores separately, the non-significant interaction between lesson type and period indicate that students performed equally well in these questions despite the lesson type (Figs. S2B and S2C; Table S6).

When analysing the variation in standardised score, the results were similar. Among all question types, there was a lack of interaction between lesson type and period (before or after) on the proportion of quiz answers being correct (Table S7). Also, when examining only reproduction-type questions, there was a significant interaction between lesson type and period on the proportion of quiz answers being correct for 2015 students (χ²₁ = 6.23, p = 0.044; Table S7).