mlsanityData to 14 Oct 2026

Large Language Models (GPT) for automating feedback on programming assignments

arXiv 2307.00150 ↗First posted 30 Jun 2023cs.HC cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

Addressing the challenge of generating personalized feedback for programming assignments is demanding due to several factors, like the complexity of code syntax or different ways to correctly solve a task. In this experimental study, we automated the process of feedback generation by employing OpenAI's GPT-3.5 model to generate personalized hints for students solving programming assignments on an automated assessment platform. Students rated the usefulness of GPT-generated hints positively. The experimental group (with GPT hints enabled) relied less on the platform's regular feedback but performed better in terms of percentage of successful submissions across consecutive attempts for tasks, where GPT hints were enabled. For tasks where the GPT feedback was made unavailable, the experimental group needed significantly less time to solve assignments. Furthermore, when GPT hints were unavailable, students in the experimental condition were initially less likely to solve the assignment correctly. This suggests potential over-reliance on GPT-generated feedback. However, students in the experimental condition were able to correct reasonably rapidly, reaching the same percentage correct after seven submission attempts. The availability of GPT hints did not significantly impact students' affective state.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 3 of its 19 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

3 of 19 references.

References · 3

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Extracting Training Data from Large Language Modelscited by 2,189
  3. 2023Learning gain differences between ChatGPT and human tutor generated algebra hintscited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 19; 2025: 5; 2026: 3.

32 papers.

Sort:

By area:

  1. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 86
  2. 2024Large Language Models for Education: A Surveycited by 34
  3. 2023Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validationcited by 29
  4. 2025Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubricscited by 29
  5. 2023Exploring the Potential of Large Language Models to Generate Formative Programming Feedbackcited by 28
  6. 2024Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judgecited by 20
  7. 2024ACE: A LLM-based Negotiation Coaching Systemcited by 13
  8. 2024Large Language Models in Computer Science Education: A Systematic Literature Reviewcited by 13
  9. 2024Evaluating and Optimizing Educational Content with Large Language Model Judgmentscited by 12
  10. 2024Are Large Language Models Good Essay Graders?cited by 11
  11. 2024Evaluating the Application of Large Language Models to Generate Feedback in Programming Educationcited by 10
  12. 2024Evaluating Language Models for Generating and Judging Programming Feedbackcited by 8
20 more
  1. 2024Grading Massive Open Online Courses Using Large Language Modelscited by 7
  2. 2023Students' Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programmingcited by 6
  3. 2024Enhancing Programming Education with ChatGPT: A Case Study on Student Perceptions and Interactions in a Python Coursecited by 6
  4. 2024MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answeringcited by 6
  5. 2024Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMscited by 4
  6. 2024Automated Feedback in Math Education: A Comparative Analysis of LLMs for Open-Ended Responsescited by 4
  7. 2023The Responsible Development of Automated Student Feedback with Generative AIcited by 3
  8. 2024Generating Feedback-Ladders for Logical Errors in Programming using Large Language Modelscited by 3
  9. 2024Benchmarking Educational Program Repaircited by 2
  10. 2024Assessing Personalized AI Mentoring with Large Language Models in the Computing Fieldcited by 2
  11. 2025Scaffolding Metacognition in Programming Education: Understanding Student-AI Interactions and Design Implicationscited by 2
  12. 2024Interactions with Prompt Problems: A New Way to Teach Programming with Large Language Modelscited by 1
  13. 2025Learning Code-Edit Embedding to Model Student Debugging Behaviorcited by 1
  14. 2025Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Toolscited by 1
  15. 2026Evaluating adaptive and generative AI-based feedback and recommendations in a knowledge-graph-integrated programming learning systemcited by 1
  16. 2024Leveraging Large Language Models for Actionable Course Evaluation Student Feedback to Lecturerscited by 0
  17. 2024The Advancement of Personalized Learning Potentially Accelerated by Generative AIcited by 0
  18. 2025Design of AI-Powered Tool for Self-Regulation Support in Programming Educationcited by 0
  19. 2026Leveraging BART to Assess CS1 C++ Programming Assignments using Rubric-based Criteriacited by 0
  20. 2026AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspectivecited by 0