mlsanityData to 7 Oct 2026

Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs

arXiv 2312.05934 ↗First posted 10 Dec 2023cs.AI cs.CL cs.LGcited by 175

Not on the map. Why?

Cited by 44 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) encapsulate a vast amount of factual information within their pre-trained weights, as evidenced by their ability to answer diverse questions across different domains. However, this knowledge is inherently limited, relying heavily on the characteristics of the training data. Consequently, using external datasets to incorporate new information or refine the capabilities of LLMs on previously seen information poses a significant challenge. In this study, we compare two common approaches: unsupervised fine-tuning and retrieval-augmented generation (RAG). We evaluate both approaches on a variety of knowledge-intensive tasks across different topics. Our findings reveal that while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge. Moreover, we find that LLMs struggle to learn new factual information through unsupervised fine-tuning, and that exposing them to numerous variations of the same fact during training could alleviate this problem.

Key facts. Cited by 175 papers in the map (data through 7 Oct 2026). Builds on 45 of its 65 references that are in the map. First posted on arXiv 10 Dec 2023.

Builds on

45 of 65 references.

References · 45

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2023Mistral 7Bcited by 3,065
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
33 more
  1. 2019Language Models as Knowledge Bases?cited by 1,922
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  6. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  9. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  10. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  11. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  12. 2021MetaICL: Learning to Learn In Contextcited by 450
  13. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  14. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  15. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  16. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  17. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  18. 2022Can language models learn from explanations in context?cited by 248
  19. 2023FinGPT: Open-Source Financial Large Language Modelscited by 244
  20. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  21. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  22. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  23. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  24. 2021Meta-learning via Language Model In-context Tuningcited by 163
  25. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  26. 2020A Survey of Knowledge-Enhanced Text Generationcited by 119
  27. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  28. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  29. 2023Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Familycited by 71
  30. 2023INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Modelscited by 62
  31. 2020Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformerscited by 57
  32. 2023Lawyer LLaMA Technical Reportcited by 54
  33. 2023A Brief Report on LawGPT 1.0: A Virtual Legal Assistant Based on GPT-3cited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 66; 2025: 63; 2026: 45.

175 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024A Survey on Large Language Models for Code Generationcited by 646
  3. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  4. 2024WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Modelscited by 90
  5. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  6. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  7. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  8. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  9. 2024Synthetic continued pretrainingcited by 56
  10. 2024KG-RAG: Bridging the Gap Between Knowledge and Creativitycited by 55
  11. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 54
  12. 2025Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Modelscited by 51
38 more
  1. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  2. 2024DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literaturecited by 50
  3. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  4. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledgecited by 49
  5. 2024Improving Retrieval for RAG based Question Answering Models on Financial Documentscited by 48
  6. 2024With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generationcited by 36
  7. 2025TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecastercited by 36
  8. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 33
  9. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  10. 2024HijackRAG: Hijacking Attacks against Retrieval-Augmented Large Language Modelscited by 27
  11. 2024Telco-RAG: Navigating the Challenges of Retrieval-Augmented Language Models for Telecommunicationscited by 26
  12. 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
  13. 2024OG-RAG: Ontology-Grounded Retrieval-Augmented Generation For Large Language Modelscited by 23
  14. 2024Efficient Knowledge Injection in LLMs via Self-Distillationcited by 22
  15. 2025How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Trainingcited by 22
  16. 2024On the Way to LLM Personalization: Learning to Remember User Conversationscited by 21
  17. 2024Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulnesscited by 17
  18. 2024Advanced Natural-based interaction for the ITAlian language: LLaMAntino-3-ANITAcited by 17
  19. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  20. 2024Mitigating Hallucinations in Large Language Models via Self-Refinement-Enhanced Knowledge Retrievalcited by 16
  21. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Modelscited by 16
  22. 2024MeMemo: On-device Retrieval Augmentation for Private and Personalized Text Generationcited by 12
  23. 2024FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?cited by 12
  24. 2025When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?cited by 12
  25. 2024Large Language Models for Social Networks: Applications, Challenges, and Solutionscited by 11
  26. 2024Graph-based Uncertainty Metrics for Long-form Language Model Outputscited by 11
  27. 2025Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructionscited by 11
  28. 2025Pre-training Limited Memory Language Models with Internal and External Knowledgecited by 11
  29. 2025Learning Facts at Scale with Active Readingcited by 11
  30. 2024Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awarenesscited by 10
  31. 2024Retrieval-Augmented Generation-based Relation Extractioncited by 9
  32. 2024Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generationcited by 9
  33. 2024Evaluating the External and Parametric Knowledge Fusion of Large Language Modelscited by 9
  34. 2025Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessonscited by 9
  35. 2025RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimizationcited by 9
  36. 2024Beyond Memorization: The Challenge of Random Memory Access in Language Modelscited by 8
  37. 2025The Multi-Round Diagnostic RAG Framework for Emulating Clinical Reasoningcited by 8
  38. 2025How new data permeates LLM knowledge and how to dilute itcited by 8

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models
  2. 2025Systematic Knowledge Injection into Large Language Models via Diverse Augmentation for Domain-Specific RAG
  3. 2024Enhancing Large Language Model Performance To Answer Questions and Extract Information More Accurately
  4. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  5. 2025Post-training an LLM for RAG? Train on Self-Generated Demonstrations
  6. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely
  7. 2025Tuning LLMs by RAG Principles: Towards LLM-native Memory
  8. 2024Investigating the performance of Retrieval-Augmented Generation and fine-tuning for the development of AI-driven knowledge-based systems