mlsanityData to 14 Oct 2026

KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality

arXiv 2506.19807 ↗First posted 24 Jun 2025cs.AI cs.CL cs.CVcited by 20

On the map: LLM reasoning → Hallucination detection and mitigation

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs), particularly slow-thinking models, often exhibit severe hallucination, outputting incorrect content due to an inability to accurately recognize knowledge boundaries during reasoning. While Reinforcement Learning (RL) can enhance complex reasoning abilities, its outcome-oriented reward mechanism often lacks factual supervision over the thinking process, further exacerbating the hallucination problem. To address the high hallucination in slow-thinking models, we propose Knowledge-enhanced RL, KnowRL. KnowRL guides models to perform fact-based slow thinking by integrating a factuality reward, based on knowledge verification, into the RL training process, helping them recognize their knowledge boundaries. KnowRL guides models to perform fact-based slow thinking by integrating a factuality reward, based on knowledge verification, into the RL training process, helping them recognize their knowledge boundaries. This targeted factual input during RL training enables the model to learn and internalize fact-based reasoning strategies. By directly rewarding adherence to facts within the reasoning steps, KnowRL fosters a more reliable thinking process. Experimental results on three hallucination evaluation datasets and two reasoning evaluation datasets demonstrate that KnowRL effectively mitigates hallucinations in slow-thinking models while maintaining their original strong reasoning capabilities. Our code is available at https://github.com/zjunlp/KnowRL.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 57 of its 94 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

57 of 94 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Are Reasoning Models More Prone to Hallucination?29 May
  2. 2025The Hallucination Tax of Reinforcement Finetuning20 May
  3. 2025Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective19 May
  4. 2025Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinking2 Jan

Other references · 53

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  10. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
38 more
  1. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  3. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  5. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  6. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  7. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  8. 2023How Language Model Hallucinations Can Snowballcited by 311
  9. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  10. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  11. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  12. 2023Fine-tuning Language Models for Factualitycited by 221
  13. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  14. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  15. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  16. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  17. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  18. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 97
  19. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 91
  20. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 87
  21. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  22. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
  23. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 66
  24. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 66
  25. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  26. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  27. 2024Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigationcited by 44
  28. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Trainingcited by 34
  29. 2025The Law of Knowledge Overshadowing: Towards Understanding, Predicting, and Preventing LLM Hallucinationcited by 23
  30. 2025O2-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answeringcited by 23
  31. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  32. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planningcited by 19
  33. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  34. 2025Enhancing LLM Reliability via Explicit Knowledge Boundary Modelingcited by 13
  35. 2025Doubly Robust Alignment for Large Language Modelscited by 12
  36. 2024On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigationcited by 9
  37. 2025Provably Efficient Online RLHF with One-Pass Reward Modelingcited by 8
  38. 2023User-Controlled Knowledge Fusion in Large Language Models: Balancing Creativity and Hallucinationcited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 13.

20 papers.

Sort:

By area:

  1. 2025Learning to Reason for Factualitycited by 24
  2. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  3. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  4. 2025Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factualitycited by 4
  5. 2025MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Modelscited by 4
  6. 2026FaithRL: Learning to Reason Faithfully through Step-Level Faithfulness Maximizationcited by 4
  7. 2026Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMscited by 4
  8. 2026Beyond Math and Code: Lightweight Corpus-Grounded Process Rewards for Factual Question Answeringcited by 4
  9. 2025When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMscited by 3
  10. 2026Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertaintycited by 2
  11. 2026UCPO: Uncertainty-Aware Policy Optimizationcited by 1
  12. 2026REIN: Bridging the Gap between Reasoning and Reliability via Reflection and Abstention Alignmentcited by 1
8 more
  1. 2025Enhancing Reliability across Short and Long-Form QA via Reinforcement Learningcited by 0
  2. 2026Only Say What You Know: Calibration-Aware Generation for Long-Form Factualitycited by 0
  3. 2026GeoFaith: A Spatio-Temporal Dual View of Faithful Chain-of-Thoughtcited by 0
  4. 2026BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigationcited by 0
  5. 2026FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervisioncited by 0
  6. 2026Hallucination Across the Reasoning Lifecycle: Interface Visibility, Causal Evidence, and Release Control in Large Reasoning Modelscited by 0
  7. 2026Rethinking Faithfulness in LLMs: A Pairwise Context-Sensitive Perspectivecited by 0
  8. 2026Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparisoncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models
  2. 2026Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models
  3. 2026KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning
  4. 2025Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
  5. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
  6. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
  7. 2025Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs
  8. 2025Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectories