The Answer Is Not the Argument
On the map: LLM reasoning → Prompting and chain-of-thought reasoning
Abstract (arXiv)
Chain-of-thought monitoring is proposed for AI oversight, yet evaluations often provide monitors with a trusted reference answer. We ask whether answer access improves verification of the reasoning or mainly supplies information about its conclusion. We collected 237 naturally generated, step-numbered solutions to 79 Humanity's Last Exam physics questions and independently labelled final-answer correctness and the first false step. Eight LLM monitors evaluated the traces with varying access to the reference answer. Certification raised mean balanced accuracy from 0.637 to 0.796, but its effect on error detection depended strongly on the conclusion: recall increased by +0.299 on wrong-answer traces, while there was no evidence of improvement on correct-answer traces containing a reasoning error (-0.083, 95% CI [-0.196, +0.030]). We then held the reasoning trace fixed in a seven-monitor certificate-congruence intervention. Replacing the true certificate with the trace's own incorrect conclusion reduced flagging by 0.659 (95% CI [0.602, 0.711]) and left flagging 0.389 below the answer-blind level. Conversely, a conflicting false certificate increased flagging of clean traces by 0.580, with 82.9% of newly flagged cases assigning the alleged error to an interior reasoning step. Trusted-answer access can therefore make monitoring appear substantially stronger because aggregate performance combines independent reasoning verification with a powerful certificate-conclusion consistency signal.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 18 references that are in the map. First posted on arXiv 31 Aug 2026.
Builds on
14 of 18 references.
Closest ancestors · 6
Same direction, newest first
- 2026MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models30 Mar
- 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety15 Jul
- 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation14 Mar
- 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan
Other references · 8
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023Let's Verify Step by Stepcited by 3,789
- 2016Concrete Problems in AI Safetycited by 1,642
- 2018AI safety via debatecited by 400
- 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
- 2026Kimi K3: Open Frontier Intelligencecited by 138
- 2026How Much Does a Reasoning Summary Reveal? An Observability Ladder for Large Language Modelscited by 2
- 2026LLM-as-a-judge validity is strongly task-dependent across physics assessment formatscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Reasoning Traces Become Performative: Step-Level Evidence that Chain-of-Thought Is an Imperfect Oversight Channel
- 2026Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluation
- 2026Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces
- 2026The Last Word Often Wins: A Format Confound in Chain-of-Thought Corruption Studies
- 2026Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation
- 2026Reasoning-Token Spikes Under Prompted Untruthful Responding in Large Language Models
- 2026A Reference-Free Score for Detecting Silent Reasoning Failures in Large Language Models
- 2026When Chain-of-Thought Fails, the Solution Hides in the Hidden States