mlsanityData to 7 Oct 2026

A Survey on Knowledge Distillation of Large Language Models

arXiv 2402.13116 ↗First posted 20 Feb 2024cs.CLcited by 276

Not on the map. Why?

Cited by 70 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the era of Large Language Models (LLMs), Knowledge Distillation (KD) emerges as a pivotal methodology for transferring advanced capabilities from leading proprietary LLMs, such as GPT-4, to their open-source counterparts like LLaMA and Mistral. Additionally, as open-source LLMs flourish, KD plays a crucial role in both compressing these models, and facilitating their self-improvement by employing themselves as teachers. This paper presents a comprehensive survey of KD's role within the realm of LLM, highlighting its critical function in imparting advanced knowledge to smaller models and its utility in model compression and self-improvement. Our survey is meticulously structured around three foundational pillars: textitalgorithm, textitskill, and textitverticalization -- providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a powerful paradigm within the KD framework to bolster LLMs' performance. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts. This work aims to provide an insightful guide for researchers and practitioners, offering a detailed overview of current methodologies in KD and proposing future research directions. Importantly, we firmly advocate for compliance with the legal terms that regulate the use of LLMs, ensuring ethical and lawful application of KD of LLMs. An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs.

Key facts. Cited by 276 papers in the map (data through 7 Oct 2026). Builds on 333 of its 388 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

333 of 388 references.

References · 333

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  13. 2020Learning to summarize from human feedbackcited by 2,562
  14. 2023A Survey of Large Language Modelscited by 2,537
  15. 2022Emergent Abilities of Large Language Modelscited by 2,483
  16. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  17. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  18. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  19. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  20. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  21. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  22. 2020Knowledge Distillation: A Surveycited by 1,578
  23. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  24. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  25. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  26. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  27. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  28. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  29. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  30. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  31. 2023LIMA: Less Is More for Alignmentcited by 1,080
  32. 2018Experience Replay for Continual Learningcited by 1,074
  33. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  34. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  35. 2021Ethical and social risks of harm from Language Modelscited by 957
  36. 2021Learning to Prompt for Continual Learningcited by 938
  37. 2023BloombergGPT: A Large Language Model for Financecited by 923
  38. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891

50 most cited of 333 All 333 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 51; 2025: 135; 2026: 90.

276 papers.

Sort:

By area:

  1. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  4. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  5. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  6. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  7. 2025Agentic Large Language Models, a surveycited by 105
  8. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 90
  9. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  10. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 61
  11. 2025Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?cited by 58
  12. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
38 more
  1. 2025Contextual Agent Security: A Policy for Every Purposecited by 55
  2. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  3. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  4. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 44
  5. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  6. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41
  7. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  8. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  9. 2024Densing Law of LLMscited by 39
  10. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 36
  11. 2025KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learningcited by 35
  12. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  13. 2024CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solvingcited by 31
  14. 2024Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initializationcited by 29
  15. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  16. 2025Large Language Models for Multi-Robot Systems: A Surveycited by 26
  17. 2025Towards Understanding Distilled Reasoning Models: A Representational Approachcited by 26
  18. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusioncited by 25
  19. 2024Language Models can Self-Lengthen to Generate Long Textscited by 24
  20. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 23
  21. 2024Can Large Language Models Improve the Adversarial Robustness of Graph Neural Networks?cited by 22
  22. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  23. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignmentcited by 22
  24. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Modelscited by 21
  25. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  26. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 19
  27. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18
  28. 2024Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generationcited by 17
  29. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
  30. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  31. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  32. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  33. 2025Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Frameworkcited by 16
  34. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  35. 2025Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulationcited by 15
  36. 2025VERDI: VLM-Embedded Reasoning for Autonomous Drivingcited by 15
  37. 2024Towards a theory of model distillationcited by 14
  38. 2024Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Reviewcited by 14

50 most cited of 276 All 276 (JSON)