# Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most

arXiv: https://arxiv.org/abs/2605.16207 · HTML: https://mlsanity.com/paper/2605.16207/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Educational agents and math problem solving, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 15 May 2026.

## Placed in

- LLM agents and tool use / [Educational agents and math problem solving](https://mlsanity.com/agents/educational-agents-and-math-problem-solving/)

## Builds on (most cited first, 21 of 21)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://mlsanity.com/paper/2305.14536/) (2023)
- [Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors](https://mlsanity.com/paper/2412.09416/) (2024)
- [Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation](https://mlsanity.com/paper/2405.11591/) (2024)
- [Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences](https://mlsanity.com/paper/2405.04645/) (2024)
- [Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation](https://mlsanity.com/paper/2310.03780/) (2023)
- [Beyond Final Answers: Evaluating Large Language Models for Math Tutoring](https://mlsanity.com/paper/2503.16460/) (2025)
- [EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework](https://mlsanity.com/paper/2504.14928/) (2025)
- [Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Study](https://mlsanity.com/paper/2504.05570/) (2025)
- [Can LLMs Reliably Simulate Human Learner Actions? A Simulation Authoring Framework for Open-Ended Learning Environments](https://mlsanity.com/paper/2410.02110/) (2024)
- [Automated Personalized Feedback Improves Learning Gains in an Intelligent Tutoring System](https://mlsanity.com/paper/2005.02431/) (2020)
- [Using Generative AI and Multi-Agents to Provide Automatic Feedback](https://mlsanity.com/paper/2411.07407/) (2024)
- [Combining Large Language Models with Tutoring System Intelligence: A Case Study in Caregiver Homework Support](https://mlsanity.com/paper/2412.11995/) (2024)
- [A Comprehensive Review of AI-based Intelligent Tutoring Systems: Applications and Challenges](https://mlsanity.com/paper/2507.18882/) (2025)
- [Investigating the Impact of Backward Strategy Learning in a Logic Tutor: Aiding Subgoal Learning towards Improved Problem Solving](https://mlsanity.com/paper/2208.04696/) (2022)
- [RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows](https://mlsanity.com/paper/2510.09021/) (2025)
- [LogicLearner: A Tool for the Guided Practice of Propositional Logic Proofs](https://mlsanity.com/paper/2503.19280/) (2025)
- [When Verification Hurts: Asymmetric Effects of Multi-Agent Feedback in Logic Proof Tutoring](https://mlsanity.com/paper/2603.27076/) (2026)
- [Enhancing a Student Productivity Model for Adaptive Problem-Solving Assistance](https://mlsanity.com/paper/2207.03025/) (2022)

## Built on by (most cited first, 2 of 2)

- [Correct Diagnosis, Better Feedback: A Symbolic-Verifier for Faithful LLM Tutoring Feedback in Logic Proofs](https://mlsanity.com/paper/2609.22553/) (2026)
- [Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop](https://mlsanity.com/paper/2610.15692/) (2026)

## Abstract (arXiv)

Effective tutoring requires distinguishing optimal, valid but suboptimal, and incorrect student solutions, a distinction central to intelligent tutoring systems (ITS) but untested for LLM-based tutors. As LLMs are increasingly explored as conversational complements to ITS, evaluating their diagnostic precision is essential. We present a benchmark of seven LLM feedback agents in propositional logic using knowledge-graph-derived ground truth across 10,836 solution--feedback pairs and three feedback conditions. Models achieved near-ceiling performance on optimal steps but systematically over-rejected valid but suboptimal reasoning and over-validated incorrect solutions, precisely where adaptive tutoring matters most. These failures persisted across models regardless of solution context, suggesting architectural rather than informational limits. Moreover, accurate diagnosis did not reliably produce pedagogically actionable feedback, revealing a gap between diagnostic judgment and instructional effectiveness. Our findings suggest that LLMs are better suited for hybrid architectures where KG-grounded models handle diagnosis while LLMs support open-ended scaffolding and dialogue.