mlsanityData to 14 Oct 2026

Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

arXiv 2405.05904 ↗First posted 9 May 2024cs.CLcited by 232

Not on the map. Why?

Cited by 27 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training. It is often conjectured that this can teach the model the behavior of hallucinating factually incorrect responses, as the model is trained to generate facts that are not grounded in its pre-existing knowledge. In this work, we study the impact of such exposure to new knowledge on the capability of the fine-tuned model to utilize its pre-existing knowledge. To this end, we design a controlled setup, focused on closed-book QA, where we vary the proportion of the fine-tuning examples that introduce new knowledge. We demonstrate that large language models struggle to acquire new factual knowledge through fine-tuning, as fine-tuning examples that introduce new knowledge are learned significantly slower than those consistent with the model's knowledge. However, we also find that as the examples with new knowledge are eventually learned, they linearly increase the model's tendency to hallucinate. Taken together, our results highlight the risk in introducing new factual knowledge through fine-tuning, and support the view that large language models mostly acquire factual knowledge through pre-training, whereas fine-tuning teaches them to use it more efficiently.

Key facts. Cited by 232 papers in the map (data through 14 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

36 of 53 references.

References · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2019Language Models as Knowledge Bases?cited by 1,936
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  9. 2023LIMA: Less Is More for Alignmentcited by 1,100
  10. 2022Mass-Editing Memory in a Transformercited by 1,001
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  12. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
24 more
  1. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  2. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  3. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  5. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  6. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  7. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  8. 2023The False Promise of Imitating Proprietary LLMscited by 263
  9. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  10. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  11. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  12. 2020Modifying Memories in Transformer Modelscited by 224
  13. 2023Fine-tuning Language Models for Factualitycited by 221
  14. 2022A Review on Language Models as Knowledge Basescited by 157
  15. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  16. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  17. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 87
  18. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  19. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 83
  20. 2024Calibrating Large Language Models with Sample Consistencycited by 73
  21. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 55
  22. 2024Learning or Self-aligning? Rethinking Instruction Fine-tuningcited by 43
  23. 2023ALCUNA: Large Language Models Meet New Knowledgecited by 35
  24. 2024Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answerscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 68; 2025: 106; 2026: 58.

232 papers.

Sort:

By area:

  1. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  2. 2025Why Language Models Hallucinatecited by 220
  3. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  4. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  5. 2025Confidence Improves Self-Consistency in LLMscited by 113
  6. 2024Learning Dynamics of LLM Finetuningcited by 95
  7. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  8. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  9. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 88
  10. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
  11. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  12. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 53
38 more
  1. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  2. 2025Steer LLM Latents for Hallucination Detectioncited by 50
  3. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  4. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  5. 2024Baichuan-Omni Technical Reportcited by 42
  6. 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 41
  7. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  8. 2025The Hallucination Tax of Reinforcement Finetuningcited by 40
  9. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 37
  10. 2024InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoningcited by 36
  11. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  12. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  13. 2024Scaling Laws for Fact Memorization of Large Language Modelscited by 33
  14. 2024Teaching Large Language Models to Express Knowledge Boundary from Their Own Signalscited by 32
  15. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 28
  16. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  17. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 25
  18. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMscited by 25
  19. 2024BeHonest: Benchmarking Honesty in Large Language Modelscited by 24
  20. 2024Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answeringcited by 24
  21. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 23
  22. 2024A Survey on the Honesty of Large Language Modelscited by 23
  23. 2025Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Trainingcited by 23
  24. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 23
  25. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 22
  26. 2025Position: Agent Should Invoke External Tools ONLY When Epistemically Necessarycited by 22
  27. 2024On the Way to LLM Personalization: Learning to Remember User Conversationscited by 21
  28. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  29. 2024LoGU: Long-form Generation with Uncertainty Expressionscited by 19
  30. 2025Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMscited by 19
  31. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  32. 2024Should We Really Edit Language Models? On the Evaluation of Edited Language Modelscited by 18
  33. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Modelscited by 18
  34. 2025RAG-Enhanced Collaborative LLM Agents for Drug Discoverycited by 18
  35. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  36. 2024Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluationcited by 16
  37. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  38. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 16

50 most cited of 232 All 232 (JSON)