# The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances

arXiv: https://arxiv.org/abs/2407.09975 · HTML: https://mlsanity.com/paper/2407.09975/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 50 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 7 of 7)

- [Recursive Partitioning for Heterogeneous Causal Effects](https://mlsanity.com/paper/1504.01132/) (2015)
- [Learning gain differences between ChatGPT and human tutor generated algebra hints](https://mlsanity.com/paper/2302.06871/) (2023)
- [The Econometrics of Randomized Experiments](https://mlsanity.com/paper/1607.00698/) (2016)
- [Reinforcement Learning Tutor Better Supported Lower Performers in a Math Task](https://mlsanity.com/paper/2304.04933/) (2023)
- [Are LLMs Useful in the Poorest Schools? TheTeacher.AI in Sierra Leone](https://mlsanity.com/paper/2310.02982/) (2023)
- [AI Teaches the Art of Elegant Coding: Timely, Fair, and Helpful Style Feedback in a Global Course](https://mlsanity.com/paper/2403.14986/) (2024)
- [A Generalized Approach to Power Analysis for Local Average Treatment Effects](https://mlsanity.com/paper/1610.08580/) (2016)

## Built on by (most cited first, 14 of 14)

- [Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise](https://mlsanity.com/paper/2410.03017/) (2024)
- [EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration](https://mlsanity.com/paper/2410.06238/) (2024)
- [Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solving](https://mlsanity.com/paper/2503.16434/) (2025)
- [Modeling Student Learning with 3.8 Million Program Traces](https://mlsanity.com/paper/2510.05056/) (2025)
- [Automatic Generation of Question Hints for Mathematics Problems using Large Language Models in Educational Technology](https://mlsanity.com/paper/2411.03495/) (2024)
- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [AI Meets the Classroom: When Do Large Language Models Harm Learning?](https://mlsanity.com/paper/2409.09047/) (2024)
- [Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact](https://mlsanity.com/paper/2603.00883/) (2026)
- [Human-AI Productivity Paradoxes: Modeling the Interplay of Skill, Effort, and AI Assistance](https://mlsanity.com/paper/2605.11350/) (2026)
- [How to Capture and Study Conversations Between Research Participants and ChatGPT: GPT for Researchers (g4r.org)](https://mlsanity.com/paper/2503.18303/) (2025)
- [Autoscoring Anticlimax: A Meta-analytic Understanding of AI's Short-answer Shortcomings and Wording Weaknesses](https://mlsanity.com/paper/2603.04820/) (2026)
- [RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studies](https://mlsanity.com/paper/2603.11001/) (2026)
- [Scalable LLM-based Coding of Dialogue in Healthcare Simulation: Balancing Coding Performance, Processing Time, and Environmental Impact](https://mlsanity.com/paper/2604.23255/) (2026)
- [Beyond Access: Guided LLM Scaffolding for Independent Learning in Undergraduate Statistics](https://mlsanity.com/paper/2606.01375/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are quickly being adopted in a wide range of learning experiences, especially via ubiquitous and broadly accessible chat interfaces like ChatGPT and Copilot. This type of interface is readily available to students and teachers around the world, yet relatively little research has been done to assess the impact of such generic tools on student learning. Coding education is an interesting test case, both because LLMs have strong performance on coding tasks, and because LLM-powered support tools are rapidly becoming part of the workflow of professional software engineers. To help understand the impact of generic LLM use on coding education, we conducted a large-scale randomized control trial with 5,831 students from 146 countries in an online coding class in which we provided some students with access to a chat interface with GPT-4. We estimate positive benefits on exam performance for adopters, the students who used the tool, but over all students, the advertisement of GPT-4 led to a significant average decrease in exam participation. We observe similar decreases in other forms of course engagement. However, this decrease is modulated by the student's country of origin. Offering access to LLMs to students from low human development index countries increased their exam participation rate on average. Our results suggest there may be promising benefits to using LLMs in an introductory coding class, but also potential harms for engagement, which makes their longer term impact on student success unclear. Our work highlights the need for additional investigations to help understand the potential impact of future adoption and integration of LLMs into classrooms.