# Context-Masked Truncated Reasoning Audits for Answer-Key Dependence in LLM Tutors

arXiv: https://arxiv.org/abs/2607.04572 · HTML: https://mlsanity.com/paper/2607.04572/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 31 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Chain-of-Thought Reasoning In The Wild Is Not Always Faithful](https://mlsanity.com/paper/2503.08679/) (2025)
- [Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2402.13950/) (2024)
- [MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors](https://mlsanity.com/paper/2502.18940/) (2025)
- [Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps](https://mlsanity.com/paper/2502.14829/) (2025)
- [Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort](https://mlsanity.com/paper/2510.01367/) (2025)
- [FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2510.04040/) (2025)
- [Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness](https://mlsanity.com/paper/2405.18915/) (2024)
- [Verifying Chain-of-Thought Reasoning via Its Computational Graph](https://mlsanity.com/paper/2510.09312/) (2025)
- [From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning](https://mlsanity.com/paper/2505.15607/) (2025)
- [A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models](https://mlsanity.com/paper/2505.23945/) (2025)
- [RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models](https://mlsanity.com/paper/2602.17053/) (2026)
- [Evaluating Answer Leakage Robustness of LLM Tutors against Adversarial Student Attacks](https://mlsanity.com/paper/2604.18660/) (2026)
- [SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems](https://mlsanity.com/paper/2603.17373/) (2026)
- [Detecting Unfaithful Chain-of-Thought via Circuit-Guided Internal-External Discrepancy](https://mlsanity.com/paper/2605.25603/) (2026)

## Abstract (arXiv)

Large language model (LLM) tutors may have access to teacher notes, answer keys, rubrics, or retrieved solutions while producing student-facing explanations. We study whether truncated reasoning probes can distinguish direct access to such private context from answer information carried by the written explanation. Using Truncated Reasoning AUC Evaluation (TRACE), we evaluate 1000 GSM8K problems under question-only, correct answer-key, and wrong answer-key contexts. When forced-answer probes retain the private key, answer-key TRACE AUC rises from 0.375 to 0.900, and the gold answer is recoverable with no explanation at all in 998 of 1000 cases. We then introduce a context-masked replay: answer-key-generated prefixes are probed under the corresponding question-only prompt. Masking reduces 10% prefix accuracy from 0.997 to 0.126 and median AUC from 0.900 to 0.375, nearly matching question-only values of 0.113 and 0.375. On 746 pairs where both explanations end correctly, the masked mean AUC difference is -0.0086 with a 95% bootstrap interval spanning zero. Wrong keys still account for 272 of 387 incorrect final responses, showing that private artifacts can influence outputs even when early-prefix evidence disappears after masking. These results establish context masking as necessary for attributing early answer availability to an explanation rather than its hidden input.