mlsanityData to 14 Oct 2026

Unfamiliar Finetuning Examples Control How Language Models Hallucinate

arXiv 2403.05612 ↗First posted 8 Mar 2024cs.LG cs.AI cs.CLcited by 87

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models are known to hallucinate when faced with unfamiliar queries, but the underlying mechanism that govern how models hallucinate are not yet fully understood. In this work, we find that unfamiliar examples in the models' finetuning data -- those that introduce concepts beyond the base model's scope of knowledge -- are crucial in shaping these errors. In particular, we find that an LLM's hallucinated predictions tend to mirror the responses associated with its unfamiliar finetuning examples. This suggests that by modifying how unfamiliar finetuning examples are supervised, we can influence a model's responses to unfamiliar queries (e.g., say ``I don't know''). We empirically validate this observation in a series of controlled experiments involving SFT, RL, and reward model finetuning on TriviaQA and MMLU. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations. We find that, while hallucinations from the reward model can significantly undermine the effectiveness of RL factuality finetuning, strategically controlling how reward models hallucinate can minimize these negative effects. Leveraging our previous observations on controlling hallucinations, we propose an approach for learning more reliable reward models, and show that they improve the efficacy of RL factuality finetuning in long-form biography and book/movie plot generation tasks.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 49 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

41 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Alignment for Honesty12 Dec
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
26 more
  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  5. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  8. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  9. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  10. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  11. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  12. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  13. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  14. 2022Prompting GPT-3 To Be Reliablecited by 260
  15. 2023Fine-tuning Language Models for Factualitycited by 221
  16. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  17. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  18. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 157
  19. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 124
  20. 2023Do Language Models Know When They're Hallucinating References?cited by 116
  21. 2023Calibrated Language Models Must Hallucinatecited by 115
  22. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 79
  23. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  24. 2023Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?cited by 62
  25. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  26. 2023Wikibio: a Semantic Resource for the Intersectional Analysis of Biographical Eventscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 35; 2025: 38; 2026: 14.

87 papers.

Sort:

By area:

  1. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  2. 2025Learning to Reason without External Rewardscited by 210
  3. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  5. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  6. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  7. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
  8. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
  9. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  10. 2024FLAME: Factuality-Aware Alignment for Large Language Modelscited by 47
  11. 2024Current state of LLM Risks and AI Guardrailscited by 46
  12. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 35
38 more
  1. 2024Teaching Large Language Models to Express Knowledge Boundary from Their Own Signalscited by 32
  2. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 32
  3. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 28
  4. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  5. 2024A Survey on the Honesty of Large Language Modelscited by 23
  6. 2024Who's Who: Large Language Models Meet Knowledge Conflicts in Practicecited by 21
  7. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20
  8. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 20
  9. 2024LoGU: Long-form Generation with Uncertainty Expressionscited by 19
  10. 2024Automatic Curriculum Expert Iteration for Reliable LLM Reasoningcited by 16
  11. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  12. 2024Improving Model Factuality with Fine-grained Critique-based Evaluatorcited by 15
  13. 2025LLM-CAS: Dynamic Neuron Perturbation for Real-Time Hallucination Correctioncited by 15
  14. 2024From Loops to Oops: Fallback Behaviors of Language Models Under Uncertaintycited by 14
  15. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 14
  16. 2025Learning Facts at Scale with Active Readingcited by 13
  17. 2024FactAlign: Long-form Factuality Alignment of Large Language Modelscited by 12
  18. 2024Quantized Delta Weight Is Safety Keepercited by 12
  19. 2025Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filteringcited by 12
  20. 2025GRAIT: Gradient-Driven Refusal-Aware Instruction Tuning for Effective Hallucination Mitigationcited by 11
  21. 2024Utilize the Flow before Stepping into the Same River Twice: Certainty Represented Knowledge Flow for Refusal-Aware Instruction Tuningcited by 10
  22. 2025Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformerscited by 10
  23. 2024On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigationcited by 9
  24. 2025ToolACE-R: Model-aware Iterative Training and Adaptive Refinement for Tool Learningcited by 9
  25. 2025Think, Prune, Train, Improve: Scaling Reasoning without Scaling Modelscited by 9
  26. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  27. 2024Continual Memorization of Factoids in Language Modelscited by 8
  28. 2024Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learningcited by 7
  29. 2025ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolutioncited by 7
  30. 2025Personalizing Student-Agent Interactions Using Log-Contextualized Retrieval-Augmented Generation (RAG)cited by 6
  31. 2025Stochastic Chameleons: Irrelevant Context Hallucinations Reveal Class-Based (Mis)Generalization in LLMscited by 6
  32. 2025A Survey on Proactive Defense Strategies Against Misinformation in Large Language Modelscited by 6
  33. 2025Structural Entropy Guided Agent for Detecting and Repairing Knowledge Deficiencies in LLMscited by 5
  34. 2025Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilizationcited by 4
  35. 2025The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factualitycited by 4
  36. 2025Exploring the Challenges and Opportunities of AI-assisted Codebase Generationcited by 4
  37. 2025A Stitch in Time Saves Nine: Proactive Self-Refinement for Language Modelscited by 4
  38. 2025ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learningcited by 4

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The Hallucination Tax of Reinforcement Finetuning
  2. 2025Hallucinate Less by Thinking More: Aspect-Based Causal Abstention for Large Language Models
  3. 2026Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations
  4. 2024Large Language Models are Null-Shot Learners
  5. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations
  6. 2026Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized Representations
  7. 2023Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?
  8. 2025Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillation