mlsanityData to 7 Oct 2026

Context-Masked Truncated Reasoning Audits for Answer-Key Dependence in LLM Tutors

arXiv 2607.04572 ↗First posted 6 Jul 2026cs.AI cs.LGcited by 0

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language model (LLM) tutors may have access to teacher notes, answer keys, rubrics, or retrieved solutions while producing student-facing explanations. We study whether truncated reasoning probes can distinguish direct access to such private context from answer information carried by the written explanation. Using Truncated Reasoning AUC Evaluation (TRACE), we evaluate 1000 GSM8K problems under question-only, correct answer-key, and wrong answer-key contexts. When forced-answer probes retain the private key, answer-key TRACE AUC rises from 0.375 to 0.900, and the gold answer is recoverable with no explanation at all in 998 of 1000 cases. We then introduce a context-masked replay: answer-key-generated prefixes are probed under the corresponding question-only prompt. Masking reduces 10% prefix accuracy from 0.997 to 0.126 and median AUC from 0.900 to 0.375, nearly matching question-only values of 0.113 and 0.375. On 746 pairs where both explanations end correctly, the masked mean AUC difference is -0.0086 with a 95% bootstrap interval spanning zero. Wrong keys still account for 272 of 387 incorrect final responses, showing that private artifacts can influence outputs even when early-prefix evidence disappears after masking. These results establish context masking as necessary for attributing early answer availability to an explanation rather than its hidden input.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 31 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

28 of 31 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  12. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
16 more
  1. 2022PAL: Program-aided Language Modelscited by 635
  2. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  3. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 166
  4. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 108
  5. 2025MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutorscited by 47
  6. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 43
  7. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effortcited by 25
  8. 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoningcited by 25
  9. 2024Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulnesscited by 20
  10. 2025Verifying Chain-of-Thought Reasoning via Its Computational Graphcited by 20
  11. 2025From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learningcited by 15
  12. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Modelscited by 12
  13. 2026RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Modelscited by 9
  14. 2026Evaluating Answer Leakage Robustness of LLM Tutors against Adversarial Student Attackscited by 5
  15. 2026SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systemscited by 3
  16. 2026Detecting Unfaithful Chain-of-Thought via Circuit-Guided Internal-External Discrepancycited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Teaching a Large Language Model Tutor to Withhold the Answer: A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior ↗
  2. 2026Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks
  3. 2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark
  4. 2026Skill-Conditioned Gated Self-Distillation for LLM Reasoning
  5. 2026Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning
  6. 2025Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Context
  8. 2026Hidden Thoughts Are Not Secret: Reasoning Trace Exposure in LLMs