# Guiding Next-Step Hint Generation Using Automated Tests

arXiv: https://arxiv.org/abs/2105.05519 · HTML: https://mlsanity.com/paper/2105.05519/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2021.

## Built on by (most cited first, 4 of 4)

- [TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions](https://mlsanity.com/paper/2403.18426/) (2024)
- [Navigating the Landscape of Hint Generation Research: From the Past to the Future](https://mlsanity.com/paper/2404.04728/) (2024)
- [Few-shot Question Generation for Personalized Feedback in Intelligent Tutoring Systems](https://mlsanity.com/paper/2206.04187/) (2022)
- [Designing and Evaluating Chain-of-Hints for Scientific Question Answering](https://mlsanity.com/paper/2510.21087/) (2025)

## Abstract (arXiv)

Learning basic programming with Scratch can be hard for novices and tutors alike: Students may not know how to advance when solving a task, teachers may face classrooms with many raised hands at a time, and the problem is exacerbated when novices are on their own in online or virtual lessons. It is therefore desirable to generate next-step hints automatically to provide individual feedback for students who are stuck, but current approaches rely on the availability of multiple hand-crafted or hand-selected sample solutions from which to draw valid hints, and have not been adapted for Scratch. Automated testing provides an opportunity to automatically select suitable candidate solutions for hint generation, even from a pool of student solutions using different solution approaches and varying in quality. In this paper we present Catnip, the first next-step hint generation approach for Scratch, which extends existing data-driven hint generation approaches with automated testing. Evaluation of Catnip on a dataset of student Scratch programs demonstrates that the generated hints point towards functional improvements, and the use of automated tests allows the hints to be better individualized for the chosen solution path.