# One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks

arXiv: https://arxiv.org/abs/2410.09268 · HTML: https://mlsanity.com/paper/2410.09268/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 13 of 13)

- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code](https://mlsanity.com/paper/2404.00971/) (2024)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [Bugs in Large Language Models Generated Code: An Empirical Study](https://mlsanity.com/paper/2403.08937/) (2024)
- [No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT](https://mlsanity.com/paper/2308.04838/) (2023)
- [Exploring the Potential of Large Language Models to Generate Formative Programming Feedback](https://mlsanity.com/paper/2309.00029/) (2023)
- [Next-Step Hint Generation for Introductory Programming Using Large Language Models](https://mlsanity.com/paper/2312.10055/) (2023)
- [Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices](https://mlsanity.com/paper/2404.02213/) (2024)
- [TaskTracker-tool: a Toolkit for Tracking of Code Snapshots and Activity Data During Solution of Programming Tasks](https://mlsanity.com/paper/2012.05085/) (2020)
- [Bridging Education and Development: IDEs as Interactive Learning Platforms](https://mlsanity.com/paper/2401.14284/) (2024)
- [Integrating ChatGPT in a Computer Science Course: Students Perceptions and Suggestions](https://mlsanity.com/paper/2402.01640/) (2023)

## Built on by (most cited first, 3 of 3)

- [Understanding Student Interaction with AI-Powered Next-Step Hints: Strategies and Challenges](https://mlsanity.com/paper/2511.06362/) (2025)
- [Enhancing Debugging Skills with AI-Powered Assistance: A Real-Time Tool for Debugging Support](https://mlsanity.com/paper/2601.02504/) (2026)
- [SCRIPT: Implementing an Intelligent Tutoring System for Programming in a German University Context](https://mlsanity.com/paper/2604.16117/) (2026)

## Abstract (arXiv)

Students often struggle with solving programming problems when learning to code, especially when they have to do it online, with one of the most common disadvantages of working online being the lack of personalized help. This help can be provided as next-step hint generation, i.e., showing a student what specific small step they need to do next to get to the correct solution. There are many ways to generate such hints, with large language models (LLMs) being among the most actively studied right now. While LLMs constitute a promising technology for providing personalized help, combining them with other techniques, such as static analysis, can significantly improve the output quality. In this work, we utilize this idea and propose a novel system to provide both textual and code hints for programming tasks. The pipeline of the proposed approach uses a chain-of-thought prompting technique and consists of three distinct steps: (1) generating subgoals - a list of actions to proceed with the task from the current student's solution, (2) generating the code to achieve the next subgoal, and (3) generating the text to describe this needed action. During the second step, we apply static analysis to the generated code to control its size and quality. The tool is implemented as a modification to the open-source JetBrains Academy plugin, supporting students in their in-IDE courses. To evaluate our approach, we propose a list of criteria for all steps in our pipeline and conduct two rounds of expert validation. Finally, we evaluate the next-step hints in a classroom with 14 students from two universities. Our results show that both forms of the hints - textual and code - were helpful for the students, and the proposed system helped them to proceed with the coding tasks.