mlsanityData to 14 Oct 2026

The Hallucination Tax of Reinforcement Finetuning

arXiv 2505.13988 ↗First posted 20 May 2025cs.CLcited by 40

On the map: LLM reasoning → Hallucination detection and mitigation

Abstract (arXiv)

Reinforcement finetuning (RFT) has become a standard approach for enhancing the reasoning capabilities of large language models (LLMs). However, its impact on model trustworthiness remains underexplored. In this work, we identify and systematically study a critical side effect of RFT, which we term the hallucination tax: a degradation in refusal behavior causing models to produce hallucinated answers to unanswerable questions confidently. To investigate this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of unanswerable math problems designed to probe models' ability to recognize an unanswerable question by reasoning from the insufficient or ambiguous information. Our results show that standard RFT training could reduce model refusal rates by more than 80%, which significantly increases model's tendency to hallucinate. We further demonstrate that incorporating just 10% SUM during RFT substantially restores appropriate refusal behavior, with minimal accuracy trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage inference-time compute to reason about their own uncertainty and knowledge boundaries, improving generalization not only to out-of-domain math problems but also to factual question answering tasks.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 26 of its 48 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

26 of 48 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  11. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  12. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
14 more
  1. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  2. 2023How Language Model Hallucinations Can Snowballcited by 311
  3. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  4. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 145
  5. 2025LIMR: Less is More for RL Scalingcited by 139
  6. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 93
  7. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 92
  8. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 49
  9. 2024Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problemcited by 43
  10. 2025Tina: Tiny Reasoning Models via LoRAcited by 25
  11. 2024Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learningcited by 21
  12. 2023On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-trainingcited by 14
  13. 2024Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluatorscited by 13
  14. 2023How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Rankingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 29.

40 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  3. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  4. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  5. 2026Experiential Reinforcement Learningcited by 22
  6. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 20
  7. 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agentscited by 16
  8. 2026Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RLcited by 14
  9. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  10. 2026Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decodingcited by 12
  11. 2026Over-Searching in Search-Augmented Large Language Modelscited by 11
  12. 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
28 more
  1. 2026When Silence Is Golden: Can LLMs Learn to Abstain in Temporal QA and Beyond?cited by 7
  2. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  3. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 6
  4. 2025Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaborationcited by 5
  5. 2025Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinationscited by 5
  6. 2025Answering the Unanswerable Is to Err Knowingly: Analyzing and Mitigating Abstention Failures in Large Reasoning Modelscited by 3
  7. 2026Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learningcited by 3
  8. 2026CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Modelscited by 2
  9. 2026KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learningcited by 2
  10. 2026Bridging the Detection-to-Abstention Gap in Reasoning Models under Insufficient Informationcited by 2
  11. 2026Scaling Participation in Modular AI Systemscited by 2
  12. 2025FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questionscited by 1
  13. 2026PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginningcited by 1
  14. 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
  15. 2026From Refuse to Richness: Rubric Rewards for Long-Form Hallucination Reinforcement Learningcited by 1
  16. 2025Enhancing Reliability across Short and Long-Form QA via Reinforcement Learningcited by 0
  17. 2026Discovering Implicit Large Language Model Alignment Objectivescited by 0
  18. 2026Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMscited by 0
  19. 2026Turning Intent into Specifications: A Benchmark and an Interactive User-Assistant Agentcited by 0
  20. 2026To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learningcited by 0
  21. 2026Mechanistic Attention Guidance for Agent Memory Refinementcited by 0
  22. 2026Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLMcited by 0
  23. 2026Ask, Condition or Abstain: Reinforcement Learning for Missing-Premise Reasoningcited by 0
  24. 2026Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAGcited by 0
  25. 2026Counterexample Generation via Per-Theorem Symbolic Verifiers: When Imitation Hurts and Reinforcement Repairscited by 0
  26. 2026Not Until the Evidence Says So: Teaching LLM Investigators When to Close a Casecited by 0
  27. 2026Hallucination Across the Reasoning Lifecycle: Interface Visibility, Causal Evidence, and Release Control in Large Reasoning Modelscited by 0
  28. 2026Knowing When Not to Answer: Cross-Domain and Multi-Turn Generalization of Latent Underspecification Signalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs
  2. 2025Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models
  3. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback
  4. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinate
  5. 2025TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluation
  6. 2025Improving the Reliability of LLMs: Combining CoT, RAG, Self-Consistency, and Self-Verification
  7. 2026Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models
  8. 2025Are Reasoning Models More Prone to Hallucination?