Objective: To investigate how generative AI, specifically large language models (LLMs), impacts learning outcomes in coding classes.

Methods:

• Study 1: Field data analysis from two university-level programming courses
• Studies 2 and 3: Controlled laboratory experiments with randomly assigned treatment (LLM access) and control groups

Key Findings:

• LLM usage has both positive and negative effects on learning outcomes
• Students benefit when using LLMs as personal tutors for explanations
• Learning is impaired when students excessively rely on LLMs to solve practice exercises
• Copy-and-paste functionality enables solution-seeking behavior, which negatively impacts learning
• Students without prior domain knowledge gain more from LLM access
• LLM use increases students' perceived learning progress beyond actual progress

Implications:

• LLMs show promising potential as learning support tools, but students must be cautious of pitfalls
• Educators should guide students on effective LLM use for learning
• LLM integration in education requires careful consideration of usage patterns

Limitations:

• Laboratory experiments may not fully reflect real-world learning environments
• Focus on short-term learning outcomes; long-term effects not studied

Future Directions:

• Investigate long-term effects of LLM use on learning
• Develop strategies to mitigate negative impacts of LLM use in education
• Explore LLM effects in other educational domains beyond coding

Title and Authors: "AI Meets the Classroom: When Does ChatGPT Harm Learning?" by Matthias Lehmann, Philipp B. Cornelius, and Fabian J. Sting

Published On: August 29, 2024 Published By: arXiv (preprint)