mlsanityData to 14 Oct 2026

Exploring the Potential of Large Language Models to Generate Formative Programming Feedback

arXiv 2309.00029 ↗First posted 31 Aug 2023cs.AI cs.CY cs.SEcited by 28

Not on the map. Why?

Abstract (arXiv)

Ever since the emergence of large language models (LLMs) and related applications, such as ChatGPT, its performance and error analysis for programming tasks have been subject to research. In this work-in-progress paper, we explore the potential of such LLMs for computing educators and learners, as we analyze the feedback it generates to a given input containing program code. In particular, we aim at (1) exploring how an LLM like ChatGPT responds to students seeking help with their introductory programming tasks, and (2) identifying feedback types in its responses. To achieve these goals, we used students' programming sequences from a dataset gathered within a CS1 course as input for ChatGPT along with questions required to elicit feedback and correct solutions. The results show that ChatGPT performs reasonably well for some of the introductory programming tasks and student errors, which means that students can potentially benefit. However, educators should provide guidance on how to use the provided feedback, as it can contain misleading information for novices.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 10 of its 29 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

10 of 29 references.

References · 10

  1. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 126
  2. 2022Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generationcited by 99
  3. 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 89
  4. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 82
  5. 2022Using Large Language Models to Enhance Programming Error Messagescited by 67
  6. 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 64
  7. 2023Exploring the Responses of Large Language Models to Beginner Programmers' Help Requestscited by 46
  8. 2023Large Language Models (GPT) for automating feedback on programming assignmentscited by 32
  9. 2022Repairing Bugs in Python Assignments Using Large Language Modelscited by 28
  10. 2023Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessmentscited by 10

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 18; 2025: 6.

28 papers.

Sort:

By area:

  1. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 86
  2. 2023Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validationcited by 29
  3. 2025Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubricscited by 29
  4. 2023Next-Step Hint Generation for Introductory Programming Using Large Language Modelscited by 20
  5. 2024Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judgecited by 20
  6. 2024Feedback-Generation for Programming Exercises With GPT-4cited by 14
  7. 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 14
  8. 2024Generative AI for Education (GAIED): Advances, Opportunities, and Challengescited by 13
  9. 2024Evaluating Contextually Personalized Programming Exercises Created with Generative AIcited by 13
  10. 2024How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Coursecited by 11
  11. 2024Evaluating the Application of Large Language Models to Generate Feedback in Programming Educationcited by 10
  12. 2023AI-enhanced Auto-correction of Programming Exercises: How Effective is GPT-3.5?cited by 8
16 more
  1. 2024Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPTcited by 8
  2. 2024Evaluating Language Models for Generating and Judging Programming Feedbackcited by 8
  3. 2024Grading Massive Open Online Courses Using Large Language Modelscited by 7
  4. 2024Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generationcited by 6
  5. 2024Knowledge of Pretrained Language Models on Surface Information of Tokenscited by 5
  6. 2024One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Taskscited by 3
  7. 2024Benchmarking Educational Program Repaircited by 2
  8. 2024Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learnerscited by 2
  9. 2025The Role of Generative AI in Software Student CollaborAItioncited by 2
  10. 2025Assessing Large Language Models for Automated Feedback Generation in Learning Programming Problem Solvingcited by 2
  11. 2024Interactions with Prompt Problems: A New Way to Teach Programming with Large Language Modelscited by 1
  12. 2024QACP: An Annotated Question Answering Dataset for Assisting Chinese Python Programming Learnerscited by 1
  13. 2024Leveraging Large Language Models to Generate Course-specific Semantically Annotated Learning Objectscited by 1
  14. 2025Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Toolscited by 1
  15. 2025Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?cited by 1
  16. 2025GenAI Voice Mode in Programming Educationcited by 0