mlsanityData to 14 Oct 2026

Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradicts

arXiv 2606.23175 ↗First posted 22 Jun 2026cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

AI scientist systems are described as tools, coauthors, or founders, but we evaluate them as if only the final answer matters. This position paper argues that outcome-only evaluation is insufficient, and that task outcome, mechanism fidelity, and epistemic honesty must be measured separately. Our evidence comes from 28 episodes of a coding agent attempting to rediscover a known particle identification observable in a Geant4 simulation, including an 8-episode probe across two additional frontier models. In 4/20 primary-model and 3/8 cross-model episodes, agents reach right-looking results through incorrect reasoning that breaks when conditions change, which we call Correct Answer, Wrong Mechanism (CAWM). Honesty and mechanism fidelity dissociate within a single agent trajectory. When given a partially misleading prior, all five agents reject the false component on evidence, yet one defends its chosen observable with physics inconsistent with its own data. In the simulation-based discovery setting studied here, coding agents prove reliable tools but unreliable scientific co-authors for open-ended claim-making, where co-author trust requires mechanism-fidelity verification they do not reliably self-apply. The failure is detectable, and we propose a lightweight test. A one-step regime-shift check needs only the agent's claim and flags the over-generalized cases. A companion recomputation flags the remaining cases when the correct observable is known. Together, these checks flag every CAWM case in this study.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 16 of its 29 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

16 of 29 references.

References · 16

  1. 2023Let's Verify Step by Stepcited by 3,789
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  4. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  6. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  7. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 422
  8. 2025Accelerating scientific discovery with Co-Scientistcited by 401
  9. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 235
  10. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 214
  11. 2026Towards End-to-End Automation of AI Researchcited by 159
  12. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
4 more
  1. 2025Kosmos: An AI Scientist for Autonomous Discoverycited by 92
  2. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 72
  3. 2025Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?cited by 32
  4. 2025The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Age of AI Agents Demands A New Scientific Paradigm To Sustain Trustworthy Science
  2. 2026AI scientists produce results without reasoning scientifically
  3. 2026Architecting Trust in Artificial Epistemic Agents
  4. 2026Can AI Agents Synthesize Scientific Conclusions?
  5. 2026TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents
  6. 2026Trust Between AI Agents: Measuring Formation, Breakage, and Recovery, with Implications for Governing Multi-Agent Systems
  7. 2026Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict
  8. 2026Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents