mlsanityData to 7 Oct 2026

FLAME: Factuality-Aware Alignment for Large Language Models

arXiv 2405.01525 ↗First posted 2 May 2024cs.CL cs.AIcited by 46

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment is a standard procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful AI assistants. We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of LLMs, and often leads to the generation of more false facts (i.e. hallucination). In this paper, we study how to make the LLM alignment process more factual, by first identifying factors that lead to hallucination in both alignment steps: supervised fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that training the LLM on new knowledge or unfamiliar texts can encourage hallucination. This makes SFT less factual as it trains on human labeled data that may be novel to the LLM. Furthermore, reward functions used in standard RL can also encourage hallucination, because it guides the LLM to provide more helpful responses on a diverse set of instructions, often preferring longer and more detailed responses. Based on these observations, we propose factuality-aware alignment, comprised of factuality-aware SFT and factuality-aware RL through direct preference optimization. Experiments show that our proposed factuality-aware alignment guides LLMs to output more factual responses while maintaining instruction-following capability.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 53 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

42 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinate8 Mar
  2. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets20 Jul
  3. 2023LIMA: Less Is More for Alignment18 May
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
24 more
  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  2. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  4. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2024Self-Rewarding Language Modelscited by 654
  7. 2023Active Retrieval Augmented Generationcited by 506
  8. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  9. 2023Augmented Language Models: a Surveycited by 398
  10. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  11. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  12. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  13. 2022Prompting GPT-3 To Be Reliablecited by 258
  14. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  15. 2023Fine-tuning Language Models for Factualitycited by 219
  16. 2023Self-Alignment with Instruction Backtranslationcited by 149
  17. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  18. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  19. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  20. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  21. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  22. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  23. 2023Human Feedback is not Gold Standardcited by 82
  24. 2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 24; 2026: 11.

46 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  3. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 114
  4. 2025Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Modelscited by 38
  5. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  6. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  7. 2025Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMscited by 22
  8. 2024A Survey on the Honesty of Large Language Modelscited by 21
  9. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20
  10. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 20
  11. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  12. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Modelscited by 17
34 more
  1. 2024Reasoning Elicitation in Language Models via Counterfactual Feedbackcited by 14
  2. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 12
  3. 2024Improving Factuality with Explicit Working Memorycited by 10
  4. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  5. 2025BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMscited by 8
  6. 2024Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Explorationcited by 7
  7. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Modelscited by 7
  8. 2026HalluHard: A Hard Multi-Turn Hallucination Benchmarkcited by 7
  9. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  10. 2026Self-Improving Pretraining: using post-trained models to pretrain better modelscited by 6
  11. 2025Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinationscited by 5
  12. 2025Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilizationcited by 4
  13. 2026Beyond Math and Code: Lightweight Corpus-Grounded Process Rewards for Factual Question Answeringcited by 4
  14. 20243DS: Medical Domain Adaptation of LLMs via Decomposed Difficulty-based Data Selectioncited by 3
  15. 2025The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factualitycited by 3
  16. 2025Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factualitycited by 3
  17. 2025MASH: Modeling Abstention via Selective Help-Seekingcited by 3
  18. 2024Beyond Under-Alignment: Atomic Preference Enhanced Factuality Tuning for Large Language Modelscited by 2
  19. 2025Evaluating book summaries from internal knowledge in Large Language Models: a cross-model and semantic consistency approachcited by 1
  20. 2025Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Modelscited by 1
  21. 2025Learning Auxiliary Tasks Improves Reference-Free Hallucination Detection in Open-Domain Long-Form Generationcited by 1
  22. 2025PrismRAG: Boosting RAG Factuality with Distractor Resilience and Strategized Reasoningcited by 1
  23. 2025LMAR: Language Model Augmented Retriever for Domain-specific Knowledge Indexingcited by 1
  24. 2026Understanding Why Language Models Hallucinate: Testing Reasoning Against Priorscited by 1
  25. 2025Hallucination Detection and Mitigation with Diffusion in Multi-Variate Time-Series Foundation Modelscited by 0
  26. 2025Mathematical Analysis of Hallucination Dynamics in Large Language Models: Uncertainty Quantification, Advanced Decoding, and Principled Mitigationcited by 0
  27. 2025Enhancing Reliability across Short and Long-Form QA via Reinforcement Learningcited by 0
  28. 2026LANCET: Neural Intervention via Structural Entropy for Mitigating Faithfulness Hallucinations in LLMscited by 0
  29. 2026PRISM: Probability Reallocation with In-Span Masking for Knowledge-Sensitive Alignmentcited by 0
  30. 2026Only Say What You Know: Calibration-Aware Generation for Long-Form Factualitycited by 0
  31. 2026Position: Retire the "Positive Backdoor" Label -- Secret Alignment Requires Strict and Systematic Evaluationcited by 0
  32. 2026Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gatingcited by 0
  33. 2026TrafficAlign: Aligning Large Language Models for Traffic Scenario Generationcited by 0
  34. 2026DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation
  2. 2026The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models
  3. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
  4. 2025The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMs
  5. 2025Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models
  6. 2026Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning
  7. 2025Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs
  8. 2025Learning to Reason for Factuality