# BugSpotter: Automated Generation of Code Debugging Exercises

arXiv: https://arxiv.org/abs/2411.14303 · HTML: https://mlsanity.com/paper/2411.14303/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 41 references that are in the map.
- First posted on arXiv 21 Nov 2024.

## Builds on (most cited first, 12 of 12)

- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors](https://mlsanity.com/paper/2306.17156/) (2023)
- [How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging](https://mlsanity.com/paper/2310.05292/) (2023)
- [Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation](https://mlsanity.com/paper/2310.03780/) (2023)
- [Synthesizing Tasks for Block-based Programming](https://mlsanity.com/paper/2006.16913/) (2020)
- [Neural Task Synthesis for Visual Programming](https://mlsanity.com/paper/2305.18342/) (2023)
- [Adaptive Scaffolding in Block-Based Programming via Synthesizing New Tasks as Pop Quizzes](https://mlsanity.com/paper/2303.16359/) (2023)
- [Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming](https://mlsanity.com/paper/2310.10690/) (2023)
- [Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation](https://mlsanity.com/paper/2406.05053/) (2024)
- [Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models](https://mlsanity.com/paper/2311.16017/) (2023)
- [Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming](https://mlsanity.com/paper/2406.09891/) (2024)

## Built on by (most cited first, 5 of 5)

- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions](https://mlsanity.com/paper/2504.18691/) (2025)
- [Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Models](https://mlsanity.com/paper/2503.04267/) (2025)
- [A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback](https://mlsanity.com/paper/2605.18073/) (2026)
- [Enhancing Debugging Skills with AI-Powered Assistance: A Real-Time Tool for Debugging Support](https://mlsanity.com/paper/2601.02504/) (2026)

## Abstract (arXiv)

Debugging is an essential skill when learning to program, yet its instruction and emphasis often vary widely across introductory courses. In the era of code-generating large language models (LLMs), the ability for students to reason about code and identify errors is increasingly important. However, students frequently resort to trial-and-error methods to resolve bugs without fully understanding the underlying issues. Developing the ability to identify and hypothesize the cause of bugs is crucial but can be time-consuming to teach effectively through traditional means. This paper introduces BugSpotter, an innovative tool that leverages an LLM to generate buggy code from a problem description and verify the synthesized bugs via a test suite. Students interact with BugSpotter by designing failing test cases, where the buggy code's output differs from the expected result as defined by the problem specification. This not only provides opportunities for students to enhance their debugging skills, but also to practice reading and understanding problem specifications. We deployed BugSpotter in a large classroom setting and compared the debugging exercises it generated to exercises hand-crafted by an instructor for the same problems. We found that the LLM-generated exercises produced by BugSpotter varied in difficulty and were well-matched to the problem specifications. Importantly, the LLM-generated exercises were comparable to those manually created by instructors with respect to student performance, suggesting that BugSpotter could be an effective and efficient aid for learning debugging.