# Why Didn't It Check? Unsupported Final Claims and Their Repair in Two Tool-Equipped Language Models

arXiv: https://arxiv.org/abs/2608.27768 · HTML: https://mlsanity.com/paper/2608.27768/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 23 of 23)

- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [Why Language Models Hallucinate](https://mlsanity.com/paper/2509.04664/) (2025)
- [AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://mlsanity.com/paper/2506.09038/) (2025)
- [Sufficient Context: A New Lens on Retrieval Augmented Generation Systems](https://mlsanity.com/paper/2411.06037/) (2024)
- [Preregistering NLP Research](https://mlsanity.com/paper/2103.06944/) (2021)
- [When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively](https://mlsanity.com/paper/2404.19705/) (2024)
- [When Agents Commit Too Soon: Diagnosing Premature Commitment in LLM Agents](https://mlsanity.com/paper/2606.22936/) (2026)
- [Decomposing LLM Self-Correction: The Accuracy-Correction Paradox and Error Depth Hypothesis](https://mlsanity.com/paper/2601.00828/) (2025)
- [Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay](https://mlsanity.com/paper/2608.19760/) (2026)
- [Understanding and Mitigating Premature Confidence for Better LLM Reasoning](https://mlsanity.com/paper/2605.24396/) (2026)
- [Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction](https://mlsanity.com/paper/2604.04325/) (2026)
- [Quantifying and Mitigating Premature Closure in Frontier LLMs](https://mlsanity.com/paper/2605.15000/) (2026)
- [InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioning](https://mlsanity.com/paper/2603.05909/) (2026)
- [Preventing Premature Commitment in Coding Agents with an Evidence-Conditioned Execution Layer](https://mlsanity.com/paper/2607.28815/) (2026)
- [When to Retrieve During Reasoning: Adaptive Retrieval for Large Reasoning Models](https://mlsanity.com/paper/2604.26649/) (2026)
- [The Self-Correction Illusion: Role Relabeling Gates Explicit Error Flagging in Large Language Models](https://mlsanity.com/paper/2606.05976/) (2026)
- [Interpretable Uncertainty for Adaptive Retrieval and Reasoning in Question Answering](https://mlsanity.com/paper/2607.07380/) (2026)
- [Correct Is Not Governed: Provenance Integrity in Agentic Workflows](https://mlsanity.com/paper/2608.12761/) (2026)
- [Does a Tool Result Carry More Authority Than Plain Text? Three Prospective Studies of False-Claim Adoption in a Synthetic Assignment Task with Claude Opus 5](https://mlsanity.com/paper/2608.14992/) (2026)

## Abstract (arXiv)

A language model with access to tools can commit to a final claim unsupported by the evidence it has seen, even when a single available tool call would resolve the uncertainty and its instructions explicitly forbid assumptions and guesses. We separate this failure into two precisely defined quantities: occurrence, how often the model makes an unsupported claim on its own, measured from the visible evidence and final claim without using the hidden correct answer; and conditional repair, how often those same naturally occurring unsupported claims are repaired when the missing evidence is supplied. On one fixed Qwen3-32B setup, 33 of 512 first responses to 256 new prompt templates ended with an unsupported established claim. We replayed each case from an exact copy of the state in which the claim occurred; within each matched replay, the alternative tool responses had the same structure and length and differed only in a one-character response code. Resolving evidence repaired 33 of 33 claims; a matched response carrying no useful information repaired 0 of 33. When the evidence supported the original answer, the model preserved 33 of 33, with no observed harm. In a separate experiment, on 64 cases where evidence was needed, an automatic checking rule added 21 evidence calls, corrected all 10 wrong unsupported claims, preserved the 11 that were correct by accident, and never changed a correct answer into a wrong one. On a fixed Gemma 4 setup using the same sampling settings, the model called the tool in all 512 first responses and never made an unsupported final claim, so conditional repair could not be measured for that setup. These results describe two local fixed model setups on two synthetic task families. They do not show how common this failure is in real-world deployments, nor that it reflects a general mechanism shared across models.