Beginners coding with AI: Prompts make the difference
Abstract
Generative AI (GenAI) has transformed the nature of programming education. GenAI assistants can perform various programming tasks such as coding and debugging. The study aims to explore the effectiveness of ChatGPT and Copilot in learning MATLAB programming. A mixed-methods approach was used to collect quantitative and qualitative data through programming exercises, a survey, and an interview. The study involved 148 pre-university students who used both GenAI assistants in four dimensions: code generation, code explanation, bug fixing, and partial code completion. A total of 78 students responded to the survey, and 21 students were involved in the follow-up interview. Data analysis revealed that (1) students who adopted two GenAI assistants performed better in programming; though the result was not statistically significant, (2) students perceived that ChatGPT outperformed Copilot in the four dimensions measured, and (3) major themes revealed the strengths and weaknesses of ChatGPT in learning programming. The findings underscore the importance of prompt engineering and the skills required as a navigator. This study suggests a wider implication of programming education, highlighting the need for teaching essential skills required for effective use of GenAI assistants, instead of drawing a definite conclusion about the evolving technologies. Program developers must adapt alongside evolving GenAI assistants, suggesting the need for versatile and sustainable education.