# Next-Step Hint Generation for Introductory Programming Using Large Language Models

arXiv: https://arxiv.org/abs/2312.10055 · HTML: https://mlsanity.com/paper/2312.10055/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 44 references that are in the map.
- First posted on arXiv 3 Dec 2023.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [Computing Education in the Era of Generative AI](https://mlsanity.com/paper/2306.02608/) (2023)
- [Exploring the Potential of Large Language Models to Generate Formative Programming Feedback](https://mlsanity.com/paper/2309.00029/) (2023)
- [Piloting Copilot, Codex, and StarCoder2: Hot Temperature, Cold Prompts, or Black Magic?](https://mlsanity.com/paper/2210.14699/) (2022)
- [The Continuous Hint Factory - Providing Hints in Vast and Sparsely Populated Edit Distance Spaces](https://mlsanity.com/paper/1708.06564/) (2017)
- [TaskTracker-tool: a Toolkit for Tracking of Code Snapshots and Activity Data During Solution of Programming Tasks](https://mlsanity.com/paper/2012.05085/) (2020)

## Built on by (most cited first, 20 of 20)

- [Feedback-Generation for Programming Exercises With GPT-4](https://mlsanity.com/paper/2403.04449/) (2024)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [Generative AI for Education (GAIED): Advances, Opportunities, and Challenges](https://mlsanity.com/paper/2402.01580/) (2024)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course](https://mlsanity.com/paper/2407.20792/) (2024)
- [Evaluating the Application of Large Language Models to Generate Feedback in Programming Education](https://mlsanity.com/paper/2403.09744/) (2024)
- [Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices](https://mlsanity.com/paper/2404.02213/) (2024)
- [Automatic Generation of Question Hints for Mathematics Problems using Large Language Models in Educational Technology](https://mlsanity.com/paper/2411.03495/) (2024)
- [Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT](https://mlsanity.com/paper/2405.19132/) (2024)
- [Evaluating Language Models for Generating and Judging Programming Feedback](https://mlsanity.com/paper/2407.04873/) (2024)
- [Simulating Students with Large Language Models: A Review of Architecture, Mechanisms, and Role Modelling in Education with Generative AI](https://mlsanity.com/paper/2511.06078/) (2025)
- [One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks](https://mlsanity.com/paper/2410.09268/) (2024)
- [Assessing Large Language Models for Automated Feedback Generation in Learning Programming Problem Solving](https://mlsanity.com/paper/2503.14630/) (2025)
- [The promise and limits of LLMs in constructing proofs and hints for logic problems in intelligent tutoring systems](https://mlsanity.com/paper/2505.04736/) (2025)
- [Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?](https://mlsanity.com/paper/2507.17258/) (2025)
- [Human-Human-AI Triadic Programming: Uncovering the Role of AI Agent and the Value of Human Partner in Collaborative Learning](https://mlsanity.com/paper/2601.12134/) (2026)
- [Exploring Teacher-Chatbot Interaction and Affect in Block-Based Programming](https://mlsanity.com/paper/2603.20211/) (2026)
- [To Google or To ChatGPT? A Comparison of CS2 Students' Information Gathering Approaches and Outcomes](https://mlsanity.com/paper/2501.11935/) (2025)
- [Understanding Student Interaction with AI-Powered Next-Step Hints: Strategies and Challenges](https://mlsanity.com/paper/2511.06362/) (2025)
- [SCRIPT: Implementing an Intelligent Tutoring System for Programming in a German University Context](https://mlsanity.com/paper/2604.16117/) (2026)

## Abstract (arXiv)

Large Language Models possess skills such as answering questions, writing essays or solving programming exercises. Since these models are easily accessible, researchers have investigated their capabilities and risks for programming education. This work explores how LLMs can contribute to programming education by supporting students with automated next-step hints. We investigate prompt practices that lead to effective next-step hints and use these insights to build our StAP-tutor. We evaluate this tutor by conducting an experiment with students, and performing expert assessments. Our findings show that most LLM-generated feedback messages describe one specific next step and are personalised to the student's code and approach. However, the hints may contain misleading information and lack sufficient detail when students approach the end of the assignment. This work demonstrates the potential for LLM-generated feedback, but further research is required to explore its practical implementation.