# Catching The Correct Answer Trap: Characterising AI Tutor Blind Spots When Analysing Student Reasoning

arXiv: https://arxiv.org/abs/2605.23925 · HTML: https://mlsanity.com/paper/2605.23925/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 10 of 10)

- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors](https://mlsanity.com/paper/2407.09136/) (2024)
- [LLMs can Find Mathematical Reasoning Mistakes by Pedagogical Chain-of-Thought](https://mlsanity.com/paper/2405.06705/) (2024)
- [Scalable Educational Question Generation with Pre-trained Language Models](https://mlsanity.com/paper/2305.07871/) (2023)
- [How Teachers Can Use Large Language Models and Bloom's Taxonomy to Create Educational Quizzes](https://mlsanity.com/paper/2401.05914/) (2024)
- [Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions](https://mlsanity.com/paper/2310.02439/) (2023)
- [A Novel Approach to Scalable and Automatic Topic-Controlled Question Generation in Education](https://mlsanity.com/paper/2501.05220/) (2025)
- [Next Token Knowledge Tracing: Exploiting Pretrained LLM Representations to Decode Student Behaviour](https://mlsanity.com/paper/2511.02599/) (2025)
- [TrueReason: An Exemplar Personalised Learning System Integrating Reasoning with Foundational Models](https://mlsanity.com/paper/2502.10411/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Correct Answer Trap: Pedagogically-Grounded Detection and Feedback for Hidden Misconceptions](https://mlsanity.com/paper/2606.23205/) (2026)

## Abstract (arXiv)

Intelligent tutoring systems increasingly provide automated feedback on student work, but robust feedback requires assessing reasoning, not only final answers. We study a failure mode we call the correct answer trap (CAT): models under-detect misconceptions when students reach a correct answer via flawed reasoning. Analysing real student responses from the Eedi mathematics platform, we show that 71% of these failures concentrate in just two question types, both sharing a common structure where flawed reasoning happens to produce the correct numerical answer. Comparing a fine-tuned T5 with a frontier large language model, we find that improved capabilities reduce but do not eliminate the problem (84% vs 57% detection accuracy). Even the best-performing model generates roughly four false alarms for every genuine detection, making stand-alone screening impractical at realistic class sizes. Our findings demonstrate that high overall accuracy can mask critical failures in reasoning assessment, and that careful analysis of student reasoning still benefits from human judgment.