# When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning

arXiv: https://arxiv.org/abs/2609.33220 · HTML: https://mlsanity.com/paper/2609.33220/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 47 of its 55 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Placed in

- LLM reasoning

## Builds on (most cited first, 30 of 47)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.24760/) (2025)
- [Language Models Learn to Mislead Humans via RLHF](https://mlsanity.com/paper/2409.12822/) (2024)
- [When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors](https://mlsanity.com/paper/2507.05246/) (2025)
- [Monitoring Monitorability](https://mlsanity.com/paper/2512.18311/) (2025)
- [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](https://mlsanity.com/paper/2506.09501/) (2025)
- [LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking](https://mlsanity.com/paper/2604.15149/) (2026)
- [Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications](https://mlsanity.com/paper/2408.05148/) (2024)
- [Emergent Misalignment is Easy, Narrow Misalignment is Hard](https://mlsanity.com/paper/2602.07852/) (2026)
- [Training LLMs for Honesty via Confessions](https://mlsanity.com/paper/2512.08093/) (2025)
- [Learning to Self-Verify Makes Language Models Better Reasoners](https://mlsanity.com/paper/2602.07594/) (2026)
- [Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models](https://mlsanity.com/paper/2507.02778/) (2025)

## Abstract (arXiv)

Outcome-based reinforcement learning can produce models with similar task performance but very different ways of communicating about their mistakes. We study failure disclosure: whether a model admits that an attempted solution failed rather than staying silent or presenting it as successful. Across repeated outcome-only GRPO training runs, failure disclosure varies far more than task accuracy. The pattern extends to a second reasoning task and stabilized PPO, persists at 7B, and also appears in an instruction-conditioned 32B setting. We also find that small floating-point and sampling differences during training can redirect reporting behavior even when the task objective and earlier training history are held fixed. Additional tests show that failure disclosure is not a single decision: Checking the answer, entering a report, and completing the admission can separate, and the weak point depends on the task and response format. Further, experiments with neutral controls show more broadly that behaviors left weakly constrained by training are especially likely to vary across runs, of which failure disclosure is an example. We can reduce variability in failure disclosure by discouraging the model from drifting from its starting policy on failed, well-formed responses. This makes reporting substantially more consistent, though its effect on task performance depends on the setting. Stable task accuracy therefore does not guarantee stable safety-relevant behavior: Researchers should measure these behaviors directly across runs and design training methods that keep them reliable.