# Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging

arXiv: https://arxiv.org/abs/2406.11709 · HTML: https://mlsanity.com/paper/2406.11709/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes](https://mlsanity.com/paper/2310.10648/) (2023)
- [The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts](https://mlsanity.com/paper/2211.11772/) (2022)
- [Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study](https://mlsanity.com/paper/2404.13414/) (2024)
- [Computationally Identifying Funneling and Focusing Questions in Classroom Discourse](https://mlsanity.com/paper/2208.04715/) (2022)
- [Can Language Models Employ the Socratic Method? Experiments with Code Debugging](https://mlsanity.com/paper/2310.03210/) (2023)
- [Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant](https://mlsanity.com/paper/2310.16984/) (2023)
- [Edu-ConvoKit: An Open-Source Library for Education Conversation Data](https://mlsanity.com/paper/2402.05111/) (2024)

## Built on by (most cited first, 14 of 14)

- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors](https://mlsanity.com/paper/2502.18940/) (2025)
- [Training Turn-by-Turn Verifiers for Dialogue Tutoring Agents: The Curious Case of LLMs as Your Coding Tutors](https://mlsanity.com/paper/2502.13311/) (2025)
- [CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation](https://mlsanity.com/paper/2504.21751/) (2025)
- [Tree-of-Debate: Multi-Persona Debate Trees Elicit Critical Thinking for Scientific Comparative Analysis](https://mlsanity.com/paper/2502.14767/) (2025)
- [Scaffolding Metacognition in Programming Education: Understanding Student-AI Interactions and Design Implications](https://mlsanity.com/paper/2511.04144/) (2025)
- [EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners](https://mlsanity.com/paper/2608.03206/) (2026)
- [EduDial: Constructing a Large-scale Multi-turn Teacher-Student Dialogue Corpus](https://mlsanity.com/paper/2510.12899/) (2025)
- [Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems](https://mlsanity.com/paper/2604.07304/) (2026)
- [PersonalPlan: Planning Multi-Agent Systems for Personalized Programming Learning](https://mlsanity.com/paper/2606.18633/) (2026)
- [Prompt Coach: An Empirical Evaluation of an Agentic Tutor for Learning Prompt Engineering in Software Development](https://mlsanity.com/paper/2607.06074/) (2026)
- [DICE: Decoupling Capability from Intervention Necessity in LLM Tutoring](https://mlsanity.com/paper/2610.04825/) (2026)
- [SocraticRepair: Planning-Guided Socratic Tutoring for Program Debugging and Misconception Repair](https://mlsanity.com/paper/2610.15204/) (2026)
- [Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop](https://mlsanity.com/paper/2610.15692/) (2026)

## Abstract (arXiv)

Socratic questioning is an effective teaching strategy, encouraging critical thinking and problem-solving. The conversational capabilities of large language models (LLMs) show great potential for providing scalable, real-time student guidance. However, current LLMs often give away solutions directly, making them ineffective instructors. We tackle this issue in the code debugging domain with TreeInstruct, an Instructor agent guided by a novel state space-based planning algorithm. TreeInstruct asks probing questions to help students independently identify and resolve errors. It estimates a student's conceptual and syntactical knowledge to dynamically construct a question tree based on their responses and current knowledge state, effectively addressing both independent and dependent mistakes concurrently in a multi-turn interaction setting. In addition to using an existing single-bug debugging benchmark, we construct a more challenging multi-bug dataset of 150 coding problems, incorrect solutions, and bug fixes -- all carefully constructed and annotated by experts. Extensive evaluation shows TreeInstruct's state-of-the-art performance on both datasets, proving it to be a more effective instructor than baselines. Furthermore, a real-world case study with five students of varying skill levels further demonstrates TreeInstruct's ability to guide students to debug their code efficiently with minimal turns and highly Socratic questioning.