# Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models

arXiv: https://arxiv.org/abs/2302.04662 · HTML: https://mlsanity.com/paper/2302.04662/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 24 Jan 2023.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues](https://mlsanity.com/paper/2205.07540/) (2022)
- [Automated Repair of Programs from Large Language Models](https://mlsanity.com/paper/2205.10583/) (2022)
- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Repairing Bugs in Python Assignments Using Large Language Models](https://mlsanity.com/paper/2209.14876/) (2022)
- [Automated Clustering and Program Repair for Introductory Programming Assignments](https://mlsanity.com/paper/1603.03165/) (2016)
- [Program Synthesis with Pragmatic Communication](https://mlsanity.com/paper/2007.05060/) (2020)
- [Neurosymbolic Repair for Low-Code Formula Languages](https://mlsanity.com/paper/2207.11765/) (2022)
- [Balancing Cost and Quality: An Exploration of Human-in-the-loop Frameworks for Automated Short Answer Scoring](https://mlsanity.com/paper/2206.08288/) (2022)

## Built on by (most cited first, 30 of 34)

- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions](https://mlsanity.com/paper/2312.12450/) (2023)
- [Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs](https://mlsanity.com/paper/2308.09895/) (2023)
- [Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors](https://mlsanity.com/paper/2306.17156/) (2023)
- [Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation](https://mlsanity.com/paper/2310.03780/) (2023)
- [Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge](https://mlsanity.com/paper/2405.05253/) (2024)
- [Harnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectives](https://mlsanity.com/paper/2306.17459/) (2023)
- [Neural Task Synthesis for Visual Programming](https://mlsanity.com/paper/2305.18342/) (2023)
- [Evaluating ChatGPT and GPT-4 for Visual Programming](https://mlsanity.com/paper/2308.02522/) (2023)
- [Generative AI for Education (GAIED): Advances, Opportunities, and Challenges](https://mlsanity.com/paper/2402.01580/) (2024)
- [A review on the use of large language models as virtual tutors](https://mlsanity.com/paper/2405.11983/) (2024)
- [How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course](https://mlsanity.com/paper/2407.20792/) (2024)
- [Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming](https://mlsanity.com/paper/2310.10690/) (2023)
- [AI-enhanced Auto-correction of Programming Exercises: How Effective is GPT-3.5?](https://mlsanity.com/paper/2311.10737/) (2023)
- [Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT](https://mlsanity.com/paper/2405.19132/) (2024)
- [Evaluating Language Models for Generating and Judging Programming Feedback](https://mlsanity.com/paper/2407.04873/) (2024)
- [Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation](https://mlsanity.com/paper/2406.05053/) (2024)
- [Enhancing Programming Education with ChatGPT: A Case Study on Student Perceptions and Interactions in a Python Course](https://mlsanity.com/paper/2403.15472/) (2024)
- [From First Draft to Final Insight: A Multi-Agent Approach for Feedback Generation](https://mlsanity.com/paper/2505.04869/) (2025)
- [Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment](https://mlsanity.com/paper/2406.11334/) (2024)
- [Generating Feedback-Ladders for Logical Errors in Programming using Large Language Models](https://mlsanity.com/paper/2405.00302/) (2024)
- [SteLLA: A Structured Grading System Using LLMs with RAG](https://mlsanity.com/paper/2501.09092/) (2025)
- [Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agents](https://mlsanity.com/paper/2504.07655/) (2025)
- [Benchmarking Educational Program Repair](https://mlsanity.com/paper/2405.05347/) (2024)
- [MedSimAI: Simulation and Formative Feedback Generation to Enhance Deliberate Practice in Medical Education](https://mlsanity.com/paper/2503.05793/) (2025)
- [Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education](https://mlsanity.com/paper/2505.18220/) (2025)
- [The promise and limits of LLMs in constructing proofs and hints for logic problems in intelligent tutoring systems](https://mlsanity.com/paper/2505.04736/) (2025)
- [Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?](https://mlsanity.com/paper/2507.17258/) (2025)
- [The Advancement of Personalized Learning Potentially Accelerated by Generative AI](https://mlsanity.com/paper/2412.00691/) (2024)

## Abstract (arXiv)

Large language models (LLMs), such as Codex, hold great promise in enhancing programming education by automatically generating feedback for students. We investigate using LLMs to generate feedback for fixing syntax errors in Python programs, a key scenario in introductory programming. More concretely, given a student's buggy program, our goal is to generate feedback comprising a fixed program along with a natural language explanation describing the errors/fixes, inspired by how a human tutor would give feedback. While using LLMs is promising, the critical challenge is to ensure high precision in the generated feedback, which is imperative before deploying such technology in classrooms. The main research question we study is: Can we develop LLMs-based feedback generation techniques with a tunable precision parameter, giving educators quality control over the feedback that students receive? To this end, we introduce PyFiXV, our technique to generate high-precision feedback powered by Codex. The key idea behind PyFiXV is to use a novel run-time validation mechanism to decide whether the generated feedback is suitable for sharing with the student; notably, this validation mechanism also provides a precision knob to educators. We perform an extensive evaluation using two real-world datasets of Python programs with syntax errors and show the efficacy of PyFiXV in generating high-precision feedback.