mlsanityData to 7 Oct 2026

DistiLLM: Towards Streamlined Distillation for Large Language Models

arXiv 2402.03898 ↗First posted 6 Feb 2024cs.CL cs.AI cs.LGcited by 145

Not on the map. Why?

Cited by 30 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge distillation (KD) is widely used for compressing a teacher model to a smaller student model, reducing its inference cost and memory footprint while preserving model capabilities. However, current KD methods for auto-regressive sequence models (e.g., large language models) suffer from missing a standardized objective function. Moreover, the recent use of student-generated outputs to address training-inference mismatches has significantly escalated computational costs. To tackle these issues, we introduce DistiLLM, a more effective and efficient KD framework for auto-regressive language models. DistiLLM comprises two components: (1) a novel skew Kullback-Leibler divergence loss, where we unveil and leverage its theoretical properties, and (2) an adaptive off-policy approach designed to enhance the efficiency in utilizing student-generated outputs. Extensive experiments, including instruction-following tasks, demonstrate the effectiveness of DistiLLM in building high-performing student models while achieving up to 4.3× speedup compared to recent KD methods.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

31 of 49 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
19 more
  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2016Sequence-Level Knowledge Distillationcited by 761
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  8. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  10. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  11. 2020Revisiting Fundamentals of Experience Replaycited by 171
  12. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  13. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  14. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 103
  15. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  16. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  17. 2023Tailoring Language Generation Models under Total Variation Distancecited by 27
  18. 2019Practical and Consistent Estimation of f-Divergencescited by 24
  19. 2021Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integralscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 43; 2026: 86.

145 papers.

Sort:

By area:

  1. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  2. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  3. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  4. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 46
  5. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  6. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 38
  7. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  8. 2025LLM Pretraining with Continuous Conceptscited by 33
  9. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  10. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  11. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 25
  12. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 25
38 more
  1. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusioncited by 25
  2. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
  3. 2026Visual-Advantage On-Policy Distillation for Vision-Language Modelscited by 22
  4. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 21
  5. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 19
  6. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  7. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 15
  8. 2026Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillationcited by 13
  9. 2025LLM-Oriented Token-Adaptive Knowledge Distillationcited by 12
  10. 2026Multi-Turn On-Policy Distillation with Prefix Replaycited by 12
  11. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillationcited by 12
  12. 2025Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoningcited by 11
  13. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  14. 2026On-Policy Self-Distillation without Any Supervisioncited by 11
  15. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  16. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
  17. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  18. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  19. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniquescited by 9
  20. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  21. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoningcited by 9
  22. 2025Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanationcited by 8
  23. 2025Unified Reinforcement and Imitation Learning for Vision-Language Modelscited by 8
  24. 2026Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RLcited by 8
  25. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  26. 2026On-Policy Delta Distillationcited by 8
  27. 2026Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillationcited by 8
  28. 2025MemLoRA: Distilling Expert Adapters for On-Device Memory Systemscited by 7
  29. 2026Hybrid Policy Distillation for LLMscited by 7
  30. 2026ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillationcited by 7
  31. 2026SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillationcited by 7
  32. 2025Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenationcited by 6
  33. 2025Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillationcited by 6
  34. 2026UniSD: Towards a Unified Self-Distillation Framework for Large Language Modelscited by 6
  35. 2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillationcited by 6
  36. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 6
  37. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Marginscited by 5
  38. 2025InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusioncited by 5

50 most cited of 145 All 145 (JSON)