# Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation

arXiv: https://arxiv.org/abs/2310.03780 · HTML: https://mlsanity.com/paper/2310.03780/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors](https://mlsanity.com/paper/2306.17156/) (2023)
- [Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models](https://mlsanity.com/paper/2302.04662/) (2023)
- [Large Language Models (GPT) for automating feedback on programming assignments](https://mlsanity.com/paper/2307.00150/) (2023)
- [Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?](https://mlsanity.com/paper/2306.13906/) (2023)
- [Repairing Bugs in Python Assignments Using Large Language Models](https://mlsanity.com/paper/2209.14876/) (2022)
- [Exploring the Potential of Large Language Models to Generate Formative Programming Feedback](https://mlsanity.com/paper/2309.00029/) (2023)
- [Automated Clustering and Program Repair for Introductory Programming Assignments](https://mlsanity.com/paper/1603.03165/) (2016)
- [Neural Task Synthesis for Visual Programming](https://mlsanity.com/paper/2305.18342/) (2023)
- [Evaluating ChatGPT and GPT-4 for Visual Programming](https://mlsanity.com/paper/2308.02522/) (2023)

## Built on by (most cited first, 29 of 29)

- [Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge](https://mlsanity.com/paper/2405.05253/) (2024)
- [Evaluating and Optimizing Educational Content with Large Language Model Judgments](https://mlsanity.com/paper/2403.02795/) (2024)
- [Modeling Student Learning with 3.8 Million Program Traces](https://mlsanity.com/paper/2510.05056/) (2025)
- [Automatic Generation of Question Hints for Mathematics Problems using Large Language Models in Educational Technology](https://mlsanity.com/paper/2411.03495/) (2024)
- [Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming](https://mlsanity.com/paper/2310.10690/) (2023)
- [Evaluating Language Models for Generating and Judging Programming Feedback](https://mlsanity.com/paper/2407.04873/) (2024)
- [Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation](https://mlsanity.com/paper/2406.05053/) (2024)
- [Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generation](https://mlsanity.com/paper/2405.06681/) (2024)
- [BugSpotter: Automated Generation of Code Debugging Exercises](https://mlsanity.com/paper/2411.14303/) (2024)
- [Using Large Language Models for Student-Code Guided Test Case Generation in Computer Science Education](https://mlsanity.com/paper/2402.07081/) (2024)
- [Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs](https://mlsanity.com/paper/2404.18978/) (2024)
- [Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment](https://mlsanity.com/paper/2406.11334/) (2024)
- [Automated Feedback in Math Education: A Comparative Analysis of LLMs for Open-Ended Responses](https://mlsanity.com/paper/2411.08910/) (2024)
- [Evaluating Large Language Models on the GMAT: Implications for the Future of Business Education](https://mlsanity.com/paper/2401.02985/) (2024)
- [Generating Feedback-Ladders for Logical Errors in Programming using Large Language Models](https://mlsanity.com/paper/2405.00302/) (2024)
- [Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming](https://mlsanity.com/paper/2406.09891/) (2024)
- [Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agents](https://mlsanity.com/paper/2504.07655/) (2025)
- [When Verification Hurts: Asymmetric Effects of Multi-Agent Feedback in Logic Proof Tutoring](https://mlsanity.com/paper/2603.27076/) (2026)
- [Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most](https://mlsanity.com/paper/2605.16207/) (2026)
- [Learning Code-Edit Embedding to Model Student Debugging Behavior](https://mlsanity.com/paper/2502.19407/) (2025)
- [The promise and limits of LLMs in constructing proofs and hints for logic problems in intelligent tutoring systems](https://mlsanity.com/paper/2505.04736/) (2025)
- [Teaching Language Models How to Code Like Learners: Conversational Serialization for Student Simulation](https://mlsanity.com/paper/2604.10720/) (2026)
- [Towards Just-in-Time Adaptive Feedback: Enhancing Student Learning via Knowledge-Grounded LLM](https://mlsanity.com/paper/2605.26405/) (2026)
- [Enhancing tutoring systems by leveraging tailored promptings and domain knowledge with Large Language Models](https://mlsanity.com/paper/2505.02849/) (2025)
- [ClickSight: Interpreting Student Clickstreams to Reveal Insights on Learning Strategies via LLMs](https://mlsanity.com/paper/2505.15410/) (2025)
- [From Pilots to Practices: A Scoping Review of GenAI-Enabled Personalization in Computer Science Education](https://mlsanity.com/paper/2512.20714/) (2025)
- [Stable and Privacy-Preserving Synthetic Educational Data with Empirical Marginals: A Copula-Based Approach](https://mlsanity.com/paper/2604.04195/) (2026)
- [Pedagogical Promise and Peril of AI: A Text Mining Analysis of ChatGPT Research Discussions in Programming Education](https://mlsanity.com/paper/2605.00361/) (2026)
- [An Explainable AI Assistant for Introductory Programming Education: Improving Feedback Reliability with Instructor-AI Collaboration](https://mlsanity.com/paper/2606.12425/) (2026)

## Abstract (arXiv)

Generative AI and large language models hold great promise in enhancing programming education by automatically generating individualized feedback for students. We investigate the role of generative AI models in providing human tutor-style programming hints to help students resolve errors in their buggy programs. Recent works have benchmarked state-of-the-art models for various feedback generation scenarios; however, their overall quality is still inferior to human tutors and not yet ready for real-world deployment. In this paper, we seek to push the limits of generative AI models toward providing high-quality programming hints and develop a novel technique, GPT4Hints-GPT3.5Val. As a first step, our technique leverages GPT-4 as a ``tutor'' model to generate hints -- it boosts the generative quality by using symbolic information of failing test cases and fixes in prompts. As a next step, our technique leverages GPT-3.5, a weaker model, as a ``student'' model to further validate the hint quality -- it performs an automatic quality validation by simulating the potential utility of providing this feedback. We show the efficacy of our technique via extensive evaluation using three real-world datasets of Python programs covering a variety of concepts ranging from basic algorithms to regular expressions and data analysis using pandas library.