# Large Language Models (GPT) for automating feedback on programming assignments

arXiv: https://arxiv.org/abs/2307.00150 · HTML: https://mlsanity.com/paper/2307.00150/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 19 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 3 of 3)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Learning gain differences between ChatGPT and human tutor generated algebra hints](https://mlsanity.com/paper/2302.06871/) (2023)

## Built on by (most cited first, 30 of 32)

- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Large Language Models for Education: A Survey](https://mlsanity.com/paper/2405.13001/) (2024)
- [Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation](https://mlsanity.com/paper/2310.03780/) (2023)
- [Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics](https://mlsanity.com/paper/2503.23989/) (2025)
- [Exploring the Potential of Large Language Models to Generate Formative Programming Feedback](https://mlsanity.com/paper/2309.00029/) (2023)
- [Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge](https://mlsanity.com/paper/2405.05253/) (2024)
- [ACE: A LLM-based Negotiation Coaching System](https://mlsanity.com/paper/2410.01555/) (2024)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [Evaluating and Optimizing Educational Content with Large Language Model Judgments](https://mlsanity.com/paper/2403.02795/) (2024)
- [Are Large Language Models Good Essay Graders?](https://mlsanity.com/paper/2409.13120/) (2024)
- [Evaluating the Application of Large Language Models to Generate Feedback in Programming Education](https://mlsanity.com/paper/2403.09744/) (2024)
- [Evaluating Language Models for Generating and Judging Programming Feedback](https://mlsanity.com/paper/2407.04873/) (2024)
- [Grading Massive Open Online Courses Using Large Language Models](https://mlsanity.com/paper/2406.11102/) (2024)
- [Students' Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming](https://mlsanity.com/paper/2312.11567/) (2023)
- [Enhancing Programming Education with ChatGPT: A Case Study on Student Perceptions and Interactions in a Python Course](https://mlsanity.com/paper/2403.15472/) (2024)
- [MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering](https://mlsanity.com/paper/2404.12926/) (2024)
- [Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs](https://mlsanity.com/paper/2404.18978/) (2024)
- [Automated Feedback in Math Education: A Comparative Analysis of LLMs for Open-Ended Responses](https://mlsanity.com/paper/2411.08910/) (2024)
- [The Responsible Development of Automated Student Feedback with Generative AI](https://mlsanity.com/paper/2308.15334/) (2023)
- [Generating Feedback-Ladders for Logical Errors in Programming using Large Language Models](https://mlsanity.com/paper/2405.00302/) (2024)
- [Benchmarking Educational Program Repair](https://mlsanity.com/paper/2405.05347/) (2024)
- [Assessing Personalized AI Mentoring with Large Language Models in the Computing Field](https://mlsanity.com/paper/2412.08430/) (2024)
- [Scaffolding Metacognition in Programming Education: Understanding Student-AI Interactions and Design Implications](https://mlsanity.com/paper/2511.04144/) (2025)
- [Interactions with Prompt Problems: A New Way to Teach Programming with Large Language Models](https://mlsanity.com/paper/2401.10759/) (2024)
- [Learning Code-Edit Embedding to Model Student Debugging Behavior](https://mlsanity.com/paper/2502.19407/) (2025)
- [Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools](https://mlsanity.com/paper/2507.05305/) (2025)
- [Evaluating adaptive and generative AI-based feedback and recommendations in a knowledge-graph-integrated programming learning system](https://mlsanity.com/paper/2603.24940/) (2026)
- [Leveraging Large Language Models for Actionable Course Evaluation Student Feedback to Lecturers](https://mlsanity.com/paper/2407.01274/) (2024)
- [The Advancement of Personalized Learning Potentially Accelerated by Generative AI](https://mlsanity.com/paper/2412.00691/) (2024)
- [Design of AI-Powered Tool for Self-Regulation Support in Programming Education](https://mlsanity.com/paper/2504.03068/) (2025)

## Abstract (arXiv)

Addressing the challenge of generating personalized feedback for programming assignments is demanding due to several factors, like the complexity of code syntax or different ways to correctly solve a task. In this experimental study, we automated the process of feedback generation by employing OpenAI's GPT-3.5 model to generate personalized hints for students solving programming assignments on an automated assessment platform. Students rated the usefulness of GPT-generated hints positively. The experimental group (with GPT hints enabled) relied less on the platform's regular feedback but performed better in terms of percentage of successful submissions across consecutive attempts for tasks, where GPT hints were enabled. For tasks where the GPT feedback was made unavailable, the experimental group needed significantly less time to solve assignments. Furthermore, when GPT hints were unavailable, students in the experimental condition were initially less likely to solve the assignment correctly. This suggests potential over-reliance on GPT-generated feedback. However, students in the experimental condition were able to correct reasonably rapidly, reaching the same percentage correct after seven submission attempts. The availability of GPT hints did not significantly impact students' affective state.