# Enhancing Programming Error Messages in Real Time with Generative AI

arXiv: https://arxiv.org/abs/2402.08072 · HTML: https://mlsanity.com/paper/2402.08072/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 12 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Computing Education in the Era of Generative AI](https://mlsanity.com/paper/2306.02608/) (2023)
- [Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests](https://mlsanity.com/paper/2306.05715/) (2023)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Prompt Problems: A New Programming Exercise for the Generative AI Era](https://mlsanity.com/paper/2311.05943/) (2023)
- [Dcc --help: Generating Context-Aware Compiler Error Explanations with Large Language Models](https://mlsanity.com/paper/2308.11873/) (2023)
- [Prompt Middleware: Mapping Prompts for Large Language Models to UI Affordances](https://mlsanity.com/paper/2307.01142/) (2023)

## Built on by (most cited first, 3 of 3)

- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [ChatGPT and U(X): A Rapid Review on Measuring the User Experience](https://mlsanity.com/paper/2503.15808/) (2025)
- [Towards Pedagogical LLMs with Supervised Fine Tuning for Computing Education](https://mlsanity.com/paper/2411.01765/) (2024)

## Abstract (arXiv)

Generative AI is changing the way that many disciplines are taught, including computer science. Researchers have shown that generative AI tools are capable of solving programming problems, writing extensive blocks of code, and explaining complex code in simple terms. Particular promise has been shown in using generative AI to enhance programming error messages. Both students and instructors have complained for decades that these messages are often cryptic and difficult to understand. Yet recent work has shown that students make fewer repeated errors when enhanced via GPT-4. We extend this work by implementing feedback from ChatGPT for all programs submitted to our automated assessment tool, Athene, providing help for compiler, run-time, and logic errors. Our results indicate that adding generative AI to an automated assessment tool does not necessarily make it better and that design of the interface matters greatly to the usability of the feedback that GPT-4 provided.