mlsanityData to 14 Oct 2026

Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models

arXiv 2505.24630 ↗First posted 30 May 2025cs.CL cs.AIcited by 19

On the map: LLM reasoning → Hallucination detection and mitigation

Abstract (arXiv)

Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient, entropy-induced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose Factuality-aware Step-wise Policy Optimization (FSPO), an innovative RL fine-tuning algorithm incorporating explicit factuality verification at each reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incentivizing factual correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 29 of its 46 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

29 of 46 references.

References · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2024OpenAI o1 System Cardcited by 1,244
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  11. 2024DeepSeek-V3 Technical Reportcited by 811
  12. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
17 more
  1. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  2. 2025LIMO: Less is More for Reasoningcited by 498
  3. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  4. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  5. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 241
  6. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  7. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  8. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 157
  9. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 152
  10. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
  11. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 124
  12. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  13. 2023A Survey of Reasoning with Foundation Modelscited by 70
  14. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  15. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  16. 2023Diving Deep into Modes of Fact Hallucinations in Dialogue Systemscited by 32
  17. 2025Kongzi: A Historical Large Language Model with Fact Enhancementcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 14.

19 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  3. 2025Learning to Reason for Factualitycited by 24
  4. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  5. 2026Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verificationcited by 5
  6. 2026MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucinationcited by 5
  7. 2025MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Modelscited by 4
  8. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 4
  9. 2026Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Modelscited by 4
  10. 2026Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMscited by 4
  11. 2026Beyond Math and Code: Lightweight Corpus-Grounded Process Rewards for Factual Question Answeringcited by 4
  12. 2026Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibrationcited by 2
7 more
  1. 2026Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimizationcited by 2
  2. 2026Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoningcited by 2
  3. 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
  4. 2026Evaluating Reasoning Models for Queries with Presuppositionscited by 1
  5. 2026MAVEN: Multi-Agent Verification-Elaboration Network with In-Step Epistemic Auditingcited by 1
  6. 2026BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigationcited by 0
  7. 2026Hallucination Across the Reasoning Lifecycle: Interface Visibility, Causal Evidence, and Release Control in Large Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM
  2. 2025Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs
  3. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
  4. 2025Learning to Reason for Hallucination Span Detection
  5. 2026Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models
  6. 2025Are Reasoning Models More Prone to Hallucination?
  7. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
  8. 2026DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs