Cluster and trajectory analysis of motivation in an emergency remote programming course

Emergency remote teaching and motivation

A decrease in student motivation was found during the period of the pandemic in which students switched to the ERT learning modality (Camacho-Zuñiga et al., 2021). Students experienced greater problems completing assignments on time, managing their time, organizing their studies, interacting with teachers and teaching assistants (Pelikan et al., 2021), as well as more distractions and lower levels of concentration (Hussein et al., 2020). Social and peer interaction also became a challenge for students (Pelikan et al., 2021), and in some cases led to a lack of motivation to study (Kapasia et al., 2020). Accordingly, higher levels of motivation were associated with assignments that specifically promoted social interaction (Ismailov & Ono, 2021). For example, in the case of novice programmers in a hybrid course, students were found to need a sense of belonging and connectedness to others, to maintain motivation (Lohiniva & Isomöttönen, 2021).

Emotional and mental health problems were also reported, with several studies finding experiences of stress, anxiety, depression, feelings of being overwhelmed (Camacho-Zuñiga et al., 2021) and worries about contracting COVID-19 (Aguilera-Hermida, 2020). In Latin America, one study found students to be concerned about boredom, slow internet connections, lack of privacy and reduced feedback (Collazos et al., 2021). Positive emotional and social engagement is conducive to learning (Zhu et al., 2023), so the ERT situation during COVID-19 negatively affected students’ education.

Despite these difficulties, students also reported some benefits of ERT, such as saving on commute time, decreasing costs (Hussein et al., 2020), having more time for hobbies, getting more sleep, and spending more time with their families (Aguilera-Hermida, 2020). Indeed, some studies have reported cases of students who maintained or even increased their levels of motivation, e.g., due to a sense of challenge to achieve personal goals (Rahiem, 2021).

Regarding computer science students, one study concluded that despite the advantages computer science students might be expected to have in an online environment, the difficulties for them were the same under the ERT modality as for other students (Toti & Alipour, 2021). Furthermore, the negative impact of ERT was found to be greater for students in earlier grades, and significant differences in terms of motivation to learn were reported when comparing undergraduate and graduate students.

Understanding motivation profiles

The Motivated Strategies for Learning Questionnaire (MSLQ) has been used to assess the motivations and learning strategies of students (Credé & Phillips, 2011). The present study uses the motivation section of the instrument, which consists of six scales. Although some researchers have used a variable-centered approach, i.e., analyzing each variable individually, this research uses a person-centered approach, which involves studying the person or group as a whole and measuring how they behave under the combination of certain variables (Hayenga & Corpus, 2010).

Several studies have attempted to find the most adaptive and least adaptive (or maladaptive) clusters of students, whereby the greatest academic benefits are achieved by students whose profiles consist of the best combination of motivational, learning and psychological variables (Kong & Liu, 2020). Adaptive profiles are generally considered to be those that combine high levels of intrinsic goal orientation (Liu et al., 2021), self-efficacy (Kong & Liu, 2020), task value (Liu et al., 2014), control of learning beliefs (Çebi & Güyer, 2020) and low levels of anxiety (Liu et al., 2021), which is also found in courses associated with computer science (Çebi & Güyer, 2020). There is not a consensus as to whether extrinsic goal orientation is related to more or less adaptive profiles (Çebi & Güyer, 2020). More adaptive profiles tend to be associated with better results (Broadbent & Fuller-Tyszkiewicz, 2018).

Taking gender into account, one study found that the cluster with more women experienced slightly better levels of self-regulation, but that they were less prepared for the online learning environment (Yukselturk & Top, 2013). Conversely, another study reported that women were over-represented in a cluster associated with low self-efficacy; however, this could be due to the context, which was a highly competitive, male-dominated business administration degree (Bråten & Olaussen, 2005).

While previous research has infrequently studied student movements between clusters, some research considers the transition of students from one profile to another at two points during the period of study. Movements by individuals from the most adaptive to the least adaptive profile, and vice versa, are very rare (Bråten & Olaussen, 2005; Ng, Wang & Liu, 2015). Our research considers using exactly the same profiles at three set points in the semester, enabling studying student motivation trajectories.

Study context

University and course

The Pontificia Universidad Católica de Chile (UC) is a prestigious university located in Latin America, with more than 3,500 teachers and 33,000 students (Quacquarelli Symonds (QS), 2020). Previous research has highlighted the differences during the ERT period between universities with established online programs, and those that had to unexpectedly switch to online learning with no prior experience (Teodorescu, Aivaz & Amalfi, 2022). UC belonged to this second group, albeit with some experience in the use of online classes, especially due to civil unrest in the country during the second semester of 2019, which forced many courses to conduct online activities for a brief period.

This study was carried out in the Introduction to Programming undergraduate course, a compulsory module at UC for engineering, economics, physics, statistics and astronomy students. It is also on a list of electives for students from certain programs and can be taken optionally by any other student. The course teaches 1,200 students per semester, divided into between eight and 10 groups.

Prior to the pandemic, assessments in this course consisted of coding assignments submitted through a virtual mailbox, as well as written exams. However, the course became an ERPC during the pandemic and adopted the use of online judges for all assessments, i.e., web-based applications that allow for the storage of programming problems in which, through predefined inputs and outputs, students’ code can be automatically evaluated (Yera et al., 2018).

Procedure

Data was collected through three voluntarily completed online surveys, using the SurveyMonkey software. Eligible participants were those enrolled in the Introduction to Programming course during the second academic semester of 2020 (August to December). There were 1,085 students enrolled in the course, distributed across eight groups. All enrolled students were sent e-mail invitations to answer the surveys (once per survey) through the Canvas LMS learning management system software. Students were told that it was a survey about motivation, and that those who responded would be compensated with a small number of points towards one of their grades. For this reason, as well as to filter out students who did not answer all three surveys, we asked students to provide an identity number. The Comité de Ética en Ciencias Sociales, Artes y Humanidades from the Pontificia Universidad Católica de Chile granted approval to carry out this study (Reference number 200417001).

Measures

Participants

There were 745 valid responses to survey 1, 678 valid responses to survey 2 and 727 valid responses to survey 3. The responses used for this study were only those in which respondents had provided full answers to the three surveys, in addition to having signed the consent form. Responses were also removed when participants did not provide their student ID, provided an incorrect student ID (e.g., their name, or less characters than valid IDs), or answered the survey more than once (only the first answer was considered). Complete responses were collected from 481 students (318 men, 158 women, five others; average age = 18.94, minimum age = 18 years, age SD = 2.06), i.e., 44.3% (481/1,085) of all enrolled students. Of the 481 students, 88.15% were in their first year of university, 83.58% were studying engineering, and 30.35% reported having some previous programming experience. Since each of the 481 students answered the survey three times, a universe of 1,443 responses was collected.

Analytic approach

Motivation profiles in ERT during the COVID-19 pandemic (RQ1)

We obtained four clusters. Figure 1 shows the z-score values of each of the considered dimensions: intrinsic goal orientation, extrinsic goal orientation, task value, control of learning beliefs, self-efficacy and test anxiety. The figure shows cluster 1 and cluster 2; and cluster 3 and cluster 4 to be almost mirror images of each other.

To evaluate the consistency of the MSLQ scales used in the clusters, Cronbach’s alphas were calculated for each of the three periods studied during the semester. The scales with the best consistency were self-efficacy and task value, all of the values of which were equal to or greater than 0.9. These were followed by the test anxiety and intrinsic goal orientation scales, both of which had values equal to or greater than 0.75. Finally, control of learning beliefs and extrinsic goal orientation showed less satisfactory results, with the first having an alpha lower than 0.7 at T1 while the latter had alphas lower than 0.7 at T1, T2 and T3. Table 2 summarizes the results obtained from the z-scores of the four profiles. Each cluster is described below.

Cluster 1 (19.68%) is notable due to all its dimensions being low except for test anxiety, which is the highest among all groups (z-score = 1.06). This group has the lowest scores in intrinsic goal orientation (z-score = −1.40), task-value (z-score = 1.50), control of learning beliefs (z-score = −1.48) and self-efficacy for learning and performance (z-score = −1.49). This cluster was assigned the name ‘Passive Learner’ as it represents the least adaptive profile and, in general, the lowest levels of motivation and self-efficacy.

In contrast, Cluster 2 (25.71%) is notable due to all its dimensions being high except for test anxiety, which is low. This group has the highest scores in intrinsic goal orientation (z-score = 1.42), task value (z-score = 1.29), control of learning beliefs (z-score = 1.33) and self-efficacy for learning and performance (z-score = 1.31). This cluster was assigned the label ‘Active Learner’, for having, in contrast to ‘Passive Learner’, the highest levels of motivation and self-efficacy, and therefore represents the most adaptive profile.

Cluster 3 (25.71%), provides the lowest scores in extrinsic goal orientation (z-score = −1.21) and test anxiety (z-score = −1.12). It also shows a low-moderate task value. The remaining dimensions are moderate-high. This cluster was assigned the label ‘Indifferent Learner’ because it yields moderate or low levels of motivation, in conjunction with the lowest anxiety.

Cluster 4 (28.89%) is notable due to having the highest score on extrinsic goal orientation (z-score = 1.23). Its scores on test anxiety are also high and on both task value and control learning beliefs, moderate-high. However, its intrinsic goal orientation and self-efficacy for learning and performance scores are moderate-low. This cluster was assigned the label ‘Reward-oriented Learner’, as it had the highest value in extrinsic goal orientation, in addition to high anxiety.

To characterize each of the clusters, a descriptive analysis was conducted. Table 3 shows the results of each variable in a differentiated manner, segmented by profile and total study population. Table 4 displays the statistical tests performed on each of the variables, and summarizes the main results.

Representation of trajectories (RQ2)

Each of the 481 students answered the survey three times, so we had a total of 1,443 responses. The four clusters were created from all the data. Then, we returned to our initial data, in which for each student, we had their survey answers at three points in time. The three surveys were then classified as pertaining to one of the clusters, so for each student, we had their trajectory from a first, to a second, to a final third profile.

Individual students’ motivation trajectories are shown in Fig. 2, as obtained from our data by using the Disco process mining software. It displays, as rectangles, each of the four types of student profiles. The arrows between them represent transitions between the profiles, and the numbers represent the students who followed the corresponding trajectory. The inverted triangle at the top represents the beginning of the semester, while the outgoing arcs show the number of students who began the course in each respective profile (T1). For example, 71 students began as ‘Passive Learners’, while 166 students began as ‘Indifferent Learners’. Similarly, the rectangle towards the bottom represents the end of the academic semester (T3). Its input arcs show the number of students who completed the semester in each profile—e.g., 92 students completed the semester as ‘Indifferent Learners’. Accordingly, the intermediate arcs represent the movements between profiles throughout the semester (T1 to T2 and T2 to T3), e.g., five students moved from ‘Active Learners’ to ‘Passive Learners’ during the semester. In addition, under the name of each profile is the absolute frequency of events in each cluster, i.e., the number of students that constitute a profile.

Comparison of motivational trajectories according to COVID concern (RQ3)

A comparative study was conducted between the trajectories of students with a high COVID concern z-score (greater than 0.5) and those with a low z-score (less than −0.5), based on an average taken from across T1, T2 and T3. Figure 3 shows the trajectories of students with high z-scores for COVID concern, corresponding to 166/481 cases, or 34.51% of the total population. The predominant profile here was ‘Active Learner’, as 75/166 (45.18%) students began in this cluster while 101/166 (60.84%) completed the semester in this cluster. For the ‘Passive Learner’ profile, 5/166 (3.01%) students began the semester in the cluster and 3/166 (1.80%) students ended the semester in it. Only 5/166 (3.01%) students with a high COVID concern moved towards this profile throughout the entire academic period.

Figure 4 shows the behavior of students with a low COVID concern. This segment was composed of 150/481 (31.18%) students. The ‘Passive Learner’ profile predominated, having the highest inflow of cases. Moreover, 93/150 (62.00%) students completed the semester in this profile. In the case of the ‘Active Learner’ cluster, only 12/150 (8.00%) students began the semester in this cluster and even fewer ended the semester in this cluster (5/150, 3.33%).

Student profiles

Student final grades are in line with previously identified positive correlations between intrinsic goal orientation (Broadbent & Fuller-Tyszkiewicz, 2018), self-efficacy (Valle et al., 2015), task value (Ng, 2016) and academic performance. One of the causes that could be a contributing factor to these differences between the profiles is the learning environment and psychological preparation for the ERT scenario. The ‘Active Learner’ profile showed lower levels of frustration due to the course being online, which could indicate that this profile was less affected from ERT conditions. In contrast to the ‘Passive Learner’ profile, the ‘Active Learner’ cluster had students who were more comfortable using a computer and had more previous programming experience. For the ‘Passive Learner’ cluster, this would be a disadvantage not only for being in a programming course, but also for being in an online environment.

The ‘Active Learner’ profile was the one that showed the highest levels of COVID concern. Although several studies have shown that fear of COVID is associated with adverse psychological outcomes, some previous research has also found COVID fear to have a positive impact on academic motivation (Günaydın, 2021). In that study, researchers found that students’ problem-solving skills were a positive and significant predictor of COVID-19 fear, theorizing that people who are more rational are more realistic and may, therefore, have a greater concern about the virus (Günaydın, 2021). Our research contributes to the literature by showing that the trajectories of students with high and low COVID concern are different; and that students with high COVID concern are more likely associated to more adaptive profiles, while those with low COVID concern are more likely associated to less adaptive profiles.

When comparing the ‘Indifferent Learner’ and ‘Reward-oriented Learner’ profiles, it was difficult to determine which of the two was the more adaptive. Both had fairly similar values in terms of grades and MSLQ variables but differed significantly in extrinsic goal orientation and test anxiety. Previous studies have reported varying results regarding the role of extrinsic goal orientation (Çebi & Güyer, 2020; Hayenga & Corpus, 2010), however, anxiety has generally been associated with negative motivational profiles (Ng, 2016). Regarding the descriptive variables, although no significant differences with the other clusters were detected, there are some differences: the ‘Indifferent Learner’ cluster showed a slightly higher proportion of students with previous programming experience, somewhat lower levels of frustration because the courses were online, and a marginally higher proportion of students who felt comfortable using a computer. This suggests that these students were a little more adjusted to the course and modality. Given the above, we posit that ‘Indifferent Learner’ students were more adaptive than ‘Reward-oriented Learner’ students. This research contributes to the literature by identifying these four profiles and arguing they are, from most to least adaptive: ‘Active Learner’, ‘Indifferent Learner’, ‘Reward-oriented Learner’, ‘Passive Learner’.

Student trajectories

This research has provided a method to understand how student motivation changes throughout a semester. Student trajectories showed a predominance of the ‘Indifferent Learner’ profile at the beginning of the semester. However, it was the cluster with the least number of students to complete the semester and the only one which experienced a decrease over the semester. This indicates that a significant proportion of the students initially had more moderate levels of motivation, COVID concern, and anxiety, but subsequently evolved over the course of the semester towards less conservative positions. Naturally, this evolution may be tied to the evolution of the pandemic during the time period of this study. The ‘Passive Learner’ profile was the group with the fewest students at the beginning of the course, but which experienced the highest growth. This suggests high levels of lack of motivation, increased anxiety, and a lack of COVID concern. On the other hand, the ‘Reward-oriented Learner’ profile experienced the highest absolute frequency of learners. It was the profile with the highest inflow and the second highest outflow, being a high turnover cluster. Moreover, 50.93% of all students spent at least one period of the semester in this group, while 27.02% spent two or more periods therein. This indicates that a considerable proportion of the students perceived, at some point during the semester, that their predominant motivation was extrinsic, while their test anxiety was high.

Regarding the variants of the trajectories, the most frequent were found to be those in which students remained in the same profile during the three periods. Specifically in this study, the ‘Active Learner’ profile was the most stable in terms of case inflow/outflow during the semester. The remaining principal variants were mostly cases in which the student began in one cluster but then moved to another and remained in that cluster during the second and third periods. This may suggest that the motivation of students in the first period was subject to expectations driven by uncertainty about the new modality and the pandemic. Another phenomenon was that there was almost no direct movement from the ‘Passive Learner’ to the ‘Active Learner’ profile and from the ‘Active Learner’ cluster to the ‘Passive Learner’ cluster. This may indicate that, despite the variability of movement between profiles, moving from a more adaptive to a less adaptive profile, and vice versa, requires a far more drastic change in terms of motivation, study habits and study environment, the occurrence of which is far less likely. This effect has also been found in other research that analyzed profile changes over time (Bråten & Olaussen, 2005).

Specifically in the trajectories with the highest and lowest COVID concern levels, substantial differences were obtained, especially regarding the ‘Passive Learner’ and ‘Active Learner’ profiles, showing that very different trajectories are possible albeit dependent on COVID concern.

Implications of our findings

We would like to discuss two implications of our findings, which we believe are relevant for future researchers as well as practitioners: (1) being aware of motivation fluctuations, and (2) the successful use of process mining as a method of analysis of motivation trajectories.