Objective: To investigate how generative AI, specifically GPT-4, affects human learning in high school math classes.

Methods:

• Large-scale randomized controlled trial with nearly 1,000 students across three high school grades
• Three treatment arms: GPT Base (similar to ChatGPT), GPT Tutor (with learning safeguards), and control (no AI access)
• Four 90-minute sessions comprising 15% of the math curriculum
• Performance measured on assisted practice problems and unassisted exams

Key Findings:

• GPT Base improved performance on assisted practice problems by 48%
• GPT Tutor improved performance on assisted practice problems by 127%
• GPT Base decreased performance on unassisted exams by 17%
• GPT Tutor had no significant effect on unassisted exam performance
• Students overestimated the benefits of AI on their learning

Implications:

• Generative AI can significantly enhance performance when available
• Without proper safeguards, AI can inhibit learning and skill development
• Careful implementation is needed to maintain long-term productivity and learning

Limitations:

• Study focused on high school math; effects may differ in other subjects or educational levels
• Long-term impacts on learning not assessed

Future Directions:

• Investigate how to design AI tools that enhance both performance and learning
• Explore effects in different subjects and educational contexts
• Study long-term impacts on skill development and learning outcomes

Title and Authors: "Generative AI Can Harm Learning" by Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı, and Rei Mariman

Published On: August 29, 2024 (based on the arXiv submission date) Published By: arXiv (preprint)