# AI-enhanced Auto-correction of Programming Exercises: How Effective is GPT-3.5?

arXiv: https://arxiv.org/abs/2311.10737 · HTML: https://mlsanity.com/paper/2311.10737/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 13 of 13)

- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [An Analysis of the Automatic Bug Fixing Performance of ChatGPT](https://mlsanity.com/paper/2301.08653/) (2023)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Computing Education in the Era of Generative AI](https://mlsanity.com/paper/2306.02608/) (2023)
- [Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests](https://mlsanity.com/paper/2306.05715/) (2023)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models](https://mlsanity.com/paper/2302.04662/) (2023)
- [Exploring the Potential of Large Language Models to Generate Formative Programming Feedback](https://mlsanity.com/paper/2309.00029/) (2023)
- [Can ChatGPT Pass An Introductory Level Functional Language Programming Course?](https://mlsanity.com/paper/2305.02230/) (2023)

## Built on by (most cited first, 8 of 8)

- [Feedback-Generation for Programming Exercises With GPT-4](https://mlsanity.com/paper/2403.04449/) (2024)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course](https://mlsanity.com/paper/2407.20792/) (2024)
- [Evaluating the Application of Large Language Models to Generate Feedback in Programming Education](https://mlsanity.com/paper/2403.09744/) (2024)
- [Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT](https://mlsanity.com/paper/2405.19132/) (2024)
- [Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming](https://mlsanity.com/paper/2411.09261/) (2024)
- [To Google or To ChatGPT? A Comparison of CS2 Students' Information Gathering Approaches and Outcomes](https://mlsanity.com/paper/2501.11935/) (2025)

## Abstract (arXiv)

Timely formative feedback is considered as one of the most important drivers for effective learning. Delivering timely and individualized feedback is particularly challenging in large classes in higher education. Recently Large Language Models such as GPT-3 became available to the public that showed promising results on various tasks such as code generation and code explanation. This paper investigates the potential of AI in providing personalized code correction and generating feedback. Based on existing student submissions of two different real-world assignments, the correctness of the AI-aided e-assessment as well as the characteristics such as fault localization, correctness of hints, and code style suggestions of the generated feedback are investigated. The results show that 73 % of the submissions were correctly identified as either correct or incorrect. In 59 % of these cases, GPT-3.5 also successfully generated effective and high-quality feedback. Additionally, GPT-3.5 exhibited weaknesses in its evaluation, including localization of errors that were not the actual errors, or even hallucinated errors. Implications and potential new usage scenarios are discussed.