mlsanityData to 7 Oct 2026

Efficient Knowledge Injection in LLMs via Self-Distillation

arXiv 2412.14964 ↗First posted 19 Dec 2024cs.CL cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

In many practical applications, large language models (LLMs) need to acquire new knowledge not present in their pre-training data. Efficiently leveraging this knowledge usually relies on supervised fine-tuning or retrieval-augmented generation (RAG). Although RAG has emerged as the industry standard for knowledge injection, fine-tuning has not yet achieved comparable success. This paper proposes utilizing prompt distillation, a self-distillation-based method previously explored primarily for style alignment and instruction tuning, to internalize new factual knowledge from free-form documents. Unlike prior methods, our approach requires neither larger teacher models nor structured knowledge formats. Across multiple LLM sizes and model families, we show that prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. We analyze the key factors contributing to prompt distillation's effectiveness and examine how it scales.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 55 of its 73 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

55 of 73 references.

References · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
38 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2020Extracting Training Data from Large Language Modelscited by 2,144
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  13. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  14. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  15. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  16. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  17. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  18. 2023Instruction Tuning with GPT-4cited by 676
  19. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  20. 2019On the Efficacy of Knowledge Distillationcited by 377
  21. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  22. 2023Adapting Language Models to Compress Contextscited by 350
  23. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  24. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  25. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  26. 2023The False Promise of Imitating Proprietary LLMscited by 254
  27. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  28. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  29. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  30. 2022Learning by Distilling Contextcited by 128
  31. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  32. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  33. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 80
  34. 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 56
  35. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  36. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  37. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  38. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 36

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 18.

22 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 285
  2. 2026Self-Distillation Enables Continual Learningcited by 212
  3. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  4. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  5. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  6. 2026Doc-to-LoRA: Learning to Instantly Internalize Contextscited by 36
  7. 2026Aligning Language Models from User Interactionscited by 21
  8. 2025Fine-tuning with RAG for Improving LLM Learning of New Skillscited by 5
  9. 2026ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillationcited by 5
  10. 2026Rethinking Continual Experience Internalization for Self-Evolving LLM Agentscited by 4
  11. 2025Memorization and Knowledge Injection in Gated LLMscited by 2
  12. 2026SIEVE: Sample-Efficient Parametric Learning from Natural Languagecited by 2
10 more
  1. 2026MixSD: Mixed Contextual Self-Distillation for Knowledge Injectioncited by 2
  2. 2025Learning Reasoning Rewards from Expert Demonstrations with Inverse Reinforcement Learningcited by 1
  3. 2026Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gapcited by 1
  4. 2026Masked Distillation: Internalizing the Chain-of-Thought in Language Modelscited by 1
  5. 2026KALE: Enhancing Knowledge Manipulation in Large Language Models via Knowledge-aware Learningcited by 0
  6. 2026Edit Knowledge, Not Just Facts via Multi-Step Reasoning over Background Storiescited by 0
  7. 2026Parametric Knowledge in RAG-SFT for Domain-Specific Document Generationcited by 0
  8. 2026Tuning Qwen2.5-VL to Improve Its Web Interaction Skillscited by 0
  9. 2026Context Memorization for Efficient Long Context Generationcited by 0
  10. 2026Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
  2. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  3. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  4. 2024Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval
  5. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge
  6. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
  7. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data
  8. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models