# Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices

arXiv: https://arxiv.org/abs/2404.02213 · HTML: https://mlsanity.com/paper/2404.02213/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Builds on (most cited first, 9 of 9)

- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues](https://mlsanity.com/paper/2205.07540/) (2022)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment](https://mlsanity.com/paper/2309.14049/) (2023)
- ["What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models](https://mlsanity.com/paper/2304.06597/) (2023)
- [GPTutor: a ChatGPT-powered programming tool for code explanation](https://mlsanity.com/paper/2305.01863/) (2023)
- [Next-Step Hint Generation for Introductory Programming Using Large Language Models](https://mlsanity.com/paper/2312.10055/) (2023)
- [Understanding the Effects of Using Parsons Problems to Scaffold Code Writing for Students with Varying CS Self-Efficacy Levels](https://mlsanity.com/paper/2311.18115/) (2023)

## Built on by (most cited first, 10 of 10)

- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course](https://mlsanity.com/paper/2406.06451/) (2024)
- [One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks](https://mlsanity.com/paper/2410.09268/) (2024)
- [Improving Student-AI Interaction Through Pedagogical Prompting: An Example in Computer Science Education](https://mlsanity.com/paper/2506.19107/) (2025)
- [Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?](https://mlsanity.com/paper/2507.17258/) (2025)
- [Learning to Use AI for Learning: Teaching Responsible Use of AI Chatbot to K-12 Students Through an AI Literacy Module](https://mlsanity.com/paper/2508.13962/) (2025)
- [Exploring Student Choice and the Use of Multimodal Generative AI in Programming Learning](https://mlsanity.com/paper/2510.05417/) (2025)
- [Understanding Student Interaction with AI-Powered Next-Step Hints: Strategies and Challenges](https://mlsanity.com/paper/2511.06362/) (2025)
- [WordCraft: Scaffolding the Keyword Method for L2 Vocabulary Learning with Multimodal LLMs](https://mlsanity.com/paper/2602.00762/) (2026)
- [A Risk-Adaptive and Evidence-Constrained Framework for Generative AI Feedback in Programming Education](https://mlsanity.com/paper/2609.29874/) (2026)

## Abstract (arXiv)

Recent studies have integrated large language models (LLMs) into diverse educational contexts, including providing adaptive programming hints, a type of feedback focuses on helping students move forward during problem-solving. However, most existing LLM-based hint systems are limited to one single hint type. To investigate whether and how different levels of hints can support students' problem-solving and learning, we conducted a think-aloud study with 12 novices using the LLM Hint Factory, a system providing four levels of hints from general natural language guidance to concrete code assistance, varying in format and granularity. We discovered that high-level natural language hints alone can be helpless or even misleading, especially when addressing next-step or syntax-related help requests. Adding lower-level hints, like code examples with in-line comments, can better support students. The findings open up future work on customizing help responses from content, format, and granularity levels to accurately identify and meet students' learning needs.