# Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students

arXiv: https://arxiv.org/abs/2410.09193 · HTML: https://mlsanity.com/paper/2410.09193/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 50 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 10 of 10)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation](https://mlsanity.com/paper/2405.11591/) (2024)
- [A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education](https://mlsanity.com/paper/2312.03173/) (2023)
- [Empirical Evaluation of Deep Learning Models for Knowledge Tracing: Of Hyperparameters and Metrics on Performance and Replicability](https://mlsanity.com/paper/2112.15072/) (2021)
- [Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models](https://mlsanity.com/paper/2311.16017/) (2023)

## Built on by (most cited first, 6 of 6)

- [ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation](https://mlsanity.com/paper/2507.12674/) (2025)
- [On the Opportunities of Large Language Models for Programming Process Data](https://mlsanity.com/paper/2411.00414/) (2024)
- [Teaching Language Models How to Code Like Learners: Conversational Serialization for Student Simulation](https://mlsanity.com/paper/2604.10720/) (2026)
- [Distinguishing Artificial from Authentic: Evaluating LLMs for Detecting LLM-Generated Content](https://mlsanity.com/paper/2607.20446/) (2026)
- [Simulating Students' Java Programming Errors with Large Language Models](https://mlsanity.com/paper/2606.14113/) (2026)
- [To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks](https://mlsanity.com/paper/2606.30549/) (2026)

## Abstract (arXiv)

Large language models (LLMs) present an exciting opportunity for generating synthetic classroom data. Such data could include code containing a typical distribution of errors, simulated student behaviour to address the cold start problem when developing education tools, and synthetic user data when access to authentic data is restricted due to privacy reasons. In this research paper, we conduct a comparative study examining the distribution of bugs generated by LLMs in contrast to those produced by computing students. Leveraging data from two previous large-scale analyses of student-generated bugs, we investigate whether LLMs can be coaxed to exhibit bug patterns that are similar to authentic student bugs when prompted to inject errors into code. The results suggest that unguided, LLMs do not generate plausible error distributions, and many of the generated errors are unlikely to be generated by real students. However, with guidance including descriptions of common errors and typical frequencies, LLMs can be shepherded to generate realistic distributions of errors in synthetic code.