# Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning

arXiv: https://arxiv.org/abs/2512.19920 · HTML: https://mlsanity.com/paper/2512.19920/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 22 Dec 2025.

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://mlsanity.com/paper/2504.05118/) (2025)
- [Why Language Models Hallucinate](https://mlsanity.com/paper/2509.04664/) (2025)
- [Deep Think with Confidence](https://mlsanity.com/paper/2508.15260/) (2025)
- [Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty](https://mlsanity.com/paper/2507.16806/) (2025)
- [Calibrated Language Models Must Hallucinate](https://mlsanity.com/paper/2311.14648/) (2023)
- [Confidence Improves Self-Consistency in LLMs](https://mlsanity.com/paper/2502.06233/) (2025)
- [Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing](https://mlsanity.com/paper/2309.12236/) (2023)
- [FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction](https://mlsanity.com/paper/2508.11987/) (2025)
- [Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models](https://mlsanity.com/paper/2503.02623/) (2025)
- [LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses](https://mlsanity.com/paper/2310.19208/) (2023)
- [RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning](https://mlsanity.com/paper/2505.15034/) (2025)
- [Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.13445/) (2025)
- [Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.24726/) (2025)
- [FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning](https://mlsanity.com/paper/2509.13160/) (2025)
- [Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs](https://mlsanity.com/paper/2511.04869/) (2025)
- [Confidence in the Reasoning of Large Language Models](https://mlsanity.com/paper/2412.15296/) (2024)

## Built on by (most cited first, 18 of 18)

- [AgentAbstain: Do LLM Agents Know When Not to Act?](https://mlsanity.com/paper/2607.10059/) (2026)
- [On Calibration of Large Language Models: From Response To Capability](https://mlsanity.com/paper/2602.13540/) (2026)
- [BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence](https://mlsanity.com/paper/2604.03216/) (2026)
- [A Unified Definition of Hallucination: It's The World Model, Stupid!](https://mlsanity.com/paper/2512.21577/) (2025)
- [LLMs Should Express Uncertainty Explicitly](https://mlsanity.com/paper/2604.05306/) (2026)
- [Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2605.14892/) (2026)
- [I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answering](https://mlsanity.com/paper/2604.03904/) (2026)
- [TIAR: Trajectory-Informed Advantage Reweighting for LLM Abstention Learning](https://mlsanity.com/paper/2605.25850/) (2026)
- [MACD: Model-Aware Contrastive Decoding via Counterfactual Data](https://mlsanity.com/paper/2602.01740/) (2026)
- [On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models](https://mlsanity.com/paper/2607.04332/) (2026)
- [Only Say What You Know: Calibration-Aware Generation for Long-Form Factuality](https://mlsanity.com/paper/2605.01749/) (2026)
- [When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines](https://mlsanity.com/paper/2608.07813/) (2026)
- [Can a Model Catch Its Own Hallucinations for Free?: Label-Free Doubt Signals Hold Their Own Against a Labelled Dataset for Abstention](https://mlsanity.com/paper/2608.26121/) (2026)
- [Test-time Calibration Learning for Large Language Model Reasoning](https://mlsanity.com/paper/2610.02695/) (2026)
- [Hallucination Across the Reasoning Lifecycle: Interface Visibility, Causal Evidence, and Release Control in Large Reasoning Models](https://mlsanity.com/paper/2610.05472/) (2026)
- [RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty](https://mlsanity.com/paper/2610.11352/) (2026)
- [Strong yet Controllable LLM Abstention without Learning to Abstain](https://mlsanity.com/paper/2610.16318/) (2026)
- [Training Language Models to Fail Gracefully](https://mlsanity.com/paper/2610.20401/) (2026)

## Abstract (arXiv)

LLM deployment in critical domains is currently impeded by persistent hallucinations--generating plausible but factually incorrect assertions. While scaling laws drove significant improvements in general capabilities, theoretical frameworks suggest hallucination is not merely stochastic error but a predictable statistical consequence of training objectives prioritizing mimicking data distribution over epistemic honesty. Standard RLVR paradigms, utilizing binary reward signals, inadvertently incentivize models as good test-takers rather than honest communicators, encouraging guessing whenever correctness probability exceeds zero. This paper presents an exhaustive investigation into behavioral calibration, which incentivizes models to stochastically admit uncertainty by abstaining when not confident, aligning model behavior with accuracy. Synthesizing recent advances, we propose and evaluate training interventions optimizing strictly proper scoring rules for models to output a calibrated probability of correctness. Our methods enable models to either abstain from producing a complete response or flag individual claims where uncertainty remains. Utilizing Qwen3-4B-Instruct, empirical analysis reveals behavior-calibrated reinforcement learning allows smaller models to surpass frontier models in uncertainty quantification--a transferable meta-skill decouplable from raw predictive accuracy. Trained on math reasoning tasks, our model's log-scale Accuracy-to-Hallucination Ratio gain (0.806) exceeds GPT-5's (0.207) in a challenging in-domain evaluation (BeyondAIME). Moreover, in cross-domain factual QA (SimpleQA), our 4B LLM achieves zero-shot calibration error on par with frontier models including Grok-4 and Gemini-2.5-Pro, even though its factual accuracy is much lower.