# Simulating Students' Java Programming Errors with Large Language Models

arXiv: https://arxiv.org/abs/2606.14113 · HTML: https://mlsanity.com/paper/2606.14113/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Builds on (most cited first, 12 of 12)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Simulating Classroom Education with LLM-Empowered Agents](https://mlsanity.com/paper/2406.19226/) (2024)
- [Bugs in Large Language Models Generated Code: An Empirical Study](https://mlsanity.com/paper/2403.08937/) (2024)
- [Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models](https://mlsanity.com/paper/2406.08731/) (2024)
- [Embracing Imperfection: Simulating Students with Diverse Cognitive Levels Using LLM-based Agents](https://mlsanity.com/paper/2505.19997/) (2025)
- [TutorUp: What If Your Students Were Simulated? Training Tutors to Address Engagement Challenges in Online Learning](https://mlsanity.com/paper/2502.16178/) (2025)
- [LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education](https://mlsanity.com/paper/2411.10455/) (2024)
- [Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models](https://mlsanity.com/paper/2311.16017/) (2023)
- [Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students](https://mlsanity.com/paper/2410.09193/) (2024)
- [Mapping Student-AI Interaction Dynamics in Multi-Agent Learning Environments: Supporting Personalised Learning and Reducing Performance Gaps](https://mlsanity.com/paper/2506.02993/) (2025)
- [Developing Authentic Simulated Learners for Mathematics Teacher Learning: Insights from Three Approaches with Large Language Models](https://mlsanity.com/paper/2604.04361/) (2026)

## Abstract (arXiv)

Understanding student errors in the programming is a cornerstone of programming education, yet obtaining a representative set of student errors for any newly designed task remains slow and costly, since authentic submissions only accumulate after extensive classroom deployment. This paper explores whether large language models (LLMs) can serve as scalable proxies for students by simulating realistic logical errors in code submissions. Using the CodeWorkout dataset of 74,000+ unique student Java submissions across 37 problems, we evaluate five LLMs under three mainstream prompting strategies: Input-Output (IO), Chain-of-Thought (CoT), and iterative Self-Refine. We assess performance along two key dimensions: diversity (the range of distinct error patterns) and alignment (alignment with authentic student mistakes), and examine how these vary by struggling level of programming tasks. Our quantitative findings reveal that while all models generate diverse errors, their alignment to human submissions diverges: Claude Sonnet 4 achieves the most balanced performance. In addition, we conducted a blinded expert annotation study (N = 401) comparing synthetic and authentic errors. This qualitative analysis confirms that the generated errors are functionally indistinguishable from authentic student errors. Moreover, higher-struggling-level problems elicit more diverse but less student-like errors. These results highlight trade-offs in using LLMs to simulate human learners and suggest design considerations for integrating synthetic errors into teachable agents, intelligent tutoring systems, and large-scale learning analytics.