Correct Diagnosis, Better Feedback: A Symbolic-Verifier for Faithful LLM Tutoring Feedback in Logic Proofs
Not on the map. Why?
Abstract (arXiv)
Effective LLM tutoring depends on correctly identifying the specific error in a student's reasoning before generating feedback. We study this problem in propositional-logic proof tutoring, where student actions can be checked against formal inference rules. We introduce a verifier-grounded architecture that separates diagnosis from language generation. Using 600 balanced student actions, we compare a zero-shot LLM detector, a fine-tuned detector, and a symbolic verifier. Each diagnosis is processed by shared rationale and feedback agents, isolating the effect of the initial diagnosis. The zero-shot detector achieves a macro-F1 of 0.191; fine-tuning raises this to 0.709 but retains systematic errors between structurally related classes. Rationales generally preserve the diagnosis supplied to them, showing that an incorrect diagnosis can be faithfully propagated through the pipeline. Feedback can likewise remain faithful to its rationale, non-revealing, and pedagogically appropriate while addressing the wrong error. Verifier-grounded feedback achieves the highest diagnostic correctness, and expert ratings largely uneven with the automatic feedback evaluations. These findings show that apparent feedback quality can conceal upstream diagnostic errors and that faithfulness must be evaluated separately from correctness. Our code is publicly available
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 18 Sep 2026.
Builds on
19 of 32 references.
References · 19
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
- 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 131
- 2023Faithfulness Tests for Natural Language Explanationscited by 107
- 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 84
- 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanationscited by 73
- 2025Walk the Talk? Measuring the Faithfulness of Large Language Model Explanationscited by 45
- 2024Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutorscited by 41
- 2024The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Modelscited by 29
- 2024FaithLM: Towards Faithful Explanations for Large Language Modelscited by 19
- 2024MalAlgoQA: Pedagogical Evaluation of Counterfactual Reasoning in Large Language Models and Implications for AI in Educationcited by 8
7 more
- 2025LLMs cannot spot math errors, even when allowed to peek into the solutioncited by 7
- 2025A Comprehensive Review of AI-based Intelligent Tutoring Systems: Applications and Challengescited by 6
- 2025LeanTutor: Towards a Verified AI Mathematical Proof Tutorcited by 4
- 2026From Untamed Black Box to Interpretable Pedagogical Orchestration: The Ensemble of Specialized LLMs Architecture for Adaptive Tutoringcited by 3
- 2025MathEDU: Feedback Generation on Problem-Solving Processes for Mathematical Learning Supportcited by 2
- 2026When Verification Hurts: Asymmetric Effects of Multi-Agent Feedback in Logic Proof Tutoringcited by 2
- 2026Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Mostcited by 2
Built on by
Not cited yet.