mlsanityData to 7 Oct 2026

Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality

arXiv 2509.23765 ↗First posted 28 Sep 2025cs.CL cs.AI cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Hallucination in large language models (LLMs) during long-form generation remains difficult to address under existing reinforcement learning from human feedback (RLHF) frameworks, as their preference rewards often overlook the model's own knowledge boundaries. In this paper, we propose the Knowledge-Level Consistency Reinforcement Learning Framework (KLCF), which re-examines this problem from a distribution alignment perspective. KLCF formalizes long-form factuality as a bidirectional distribution matching objective between the policy model's expressed knowledge distribution and the base model's parametric knowledge distribution: under the constraint that generation must not exceed the support set of the base knowledge, the objective maximizes coverage of high-probability facts, thereby jointly optimizing precision and recall. To achieve this, we design a Dual-Fact Alignment mechanism that approximates the recall term using a factual checklist constructed by sampling from the base model, and constrains hallucinations with a lightweight truthfulness reward model. Both components are jointly optimized and require no external retrieval throughout training. Experimental results demonstrate that KLCF consistently improves factuality metrics across multiple long-form benchmarks and model scales, effectively alleviating hallucination and over-conservatism while maintaining efficiency and scalability.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 66 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

56 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs4 Mar
  2. 2024FactAlign: Long-form Factuality Alignment of Large Language Models2 Oct
  3. 2024Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction22 May
  4. 2024FLAME: Factuality-Aware Alignment for Large Language Models2 May
  5. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation14 Feb
  6. 2023Mitigating the Alignment Tax of RLHF12 Sep
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 49

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
37 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  3. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  4. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  5. 2019ELI5: Long Form Question Answeringcited by 567
  6. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  7. 2023How Language Model Hallucinations Can Snowballcited by 310
  8. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  9. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  10. 2023Fine-tuning Language Models for Factualitycited by 219
  11. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  12. 2025Learning to Reason without External Rewardscited by 199
  13. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  14. 2024Long-form factuality in large language modelscited by 146
  15. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  16. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  17. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 114
  18. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  19. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
  20. 2025HalluLens: LLM Hallucination Benchmarkcited by 88
  21. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  22. 2025Can Large Reasoning Models Self-Train?cited by 66
  23. 2024Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answeringcited by 59
  24. 2023FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answeringcited by 51
  25. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 43
  26. 2024RATT: A Thought Structure for Coherent and Correct LLM Reasoningcited by 39
  27. 2024AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotatorscited by 28
  28. 2024DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generationcited by 22
  29. 2025Learning to Reason for Factualitycited by 22
  30. 2025Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluationcited by 20
  31. 2025Towards Effective Extraction and Evaluation of Factual Claimscited by 18
  32. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
  33. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 17
  34. 2023LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisationcited by 16
  35. 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 16
  36. 2024Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methodscited by 9
  37. 2025FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factualitycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMscited by 3
  2. 2026Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learningcited by 1
  3. 2026BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation
  2. 2026FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision
  3. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback
  4. 2026Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibration
  5. 2025Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs
  6. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
  7. 2025Enhancing Reliability across Short and Long-Form QA via Reinforcement Learning
  8. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment