mlsanityData to 14 Oct 2026

Revisiting Knowledge Distillation for Autoregressive Language Models

arXiv 2402.11890 ↗First posted 19 Feb 2024cs.CLcited by 42

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge distillation (KD) is a common approach to compress a teacher model to reduce its inference cost and memory footprint, by training a smaller student model. However, in the context of autoregressive language models (LMs), we empirically find that larger teacher LMs might dramatically result in a poorer student. In response to this problem, we conduct a series of analyses and reveal that different tokens have different teaching modes, neglecting which will lead to performance degradation. Motivated by this, we propose a simple yet effective adaptive teaching approach (ATKD) to improve the KD. The core of ATKD is to reduce rote learning and make teaching more diverse and flexible. Extensive experiments on 8 LM tasks show that, with the help of ATKD, various baseline KD methods can achieve consistent and significant performance gains (up to +3.04% average score) across all model types and sizes. More encouragingly, ATKD can improve the student model generalization effectively.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 30 of its 52 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

30 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillation27 Jul
  2. 2023Lifting the Curse of Capacity Gap in Distilling Language Models20 May
  3. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation22 Aug

Other references · 27

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  11. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  12. 2016Sequence-Level Knowledge Distillationcited by 801
15 more
  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  2. 2019Green AIcited by 627
  3. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  4. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  5. 2019On the Efficacy of Knowledge Distillationcited by 381
  6. 2023The False Promise of Imitating Proprietary LLMscited by 263
  7. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  8. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  9. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  10. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  11. 2021Sparse is Enough in Scaling Transformerscited by 86
  12. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 74
  13. 2020Understanding and Improving Lexical Choice in Non-Autoregressive Translationcited by 56
  14. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  15. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 16; 2026: 19.

42 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  3. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  4. 2024ARB-LLM: Alternating Refined Binarizations for Large Language Modelscited by 31
  5. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 22
  6. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 16
  7. 2025Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Modelscited by 12
  8. 2024SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Modelscited by 9
  9. 2025CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Draftercited by 9
  10. 2025Traversal Verification for Speculative Tree Decodingcited by 9
  11. 2026Rethinking Selective Knowledge Distillationcited by 9
  12. 2025Layer-wise dynamic rank for compressing large language modelscited by 8
30 more
  1. 2025On the Generalization vs Fidelity Paradox in Knowledge Distillationcited by 7
  2. 2025Progressive Binarization with Semi-Structured Pruning for LLMscited by 5
  3. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 5
  4. 2026DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillationcited by 5
  5. 2025Optimizing Singular Spectrum for Large Language Model Compressioncited by 4
  6. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Modelscited by 3
  7. 2025KaFT: Knowledge-aware Fine-tuning for Boosting LLMs' Domain-specific Question-Answering Performancecited by 3
  8. 2026Strong Teacher Not Needed? On Distillation in LLM Pretrainingcited by 3
  9. 2024Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQLcited by 2
  10. 2025Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directionscited by 2
  11. 2025Learning a Single Token to Replace Long System Prompts in LLMscited by 2
  12. 2026Knowledge Distillation for Visual Autoregressive Modelscited by 2
  13. 20261% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillationcited by 2
  14. 2024Self-Evolution Knowledge Distillation for LLM-based Machine Translationcited by 1
  15. 2025Targeted Distillation for Sentiment Analysiscited by 1
  16. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  17. 2026TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Modelscited by 1
  18. 2025Beyond Uniform SVD:Dual-Level Optimization across Columns and Modules for LLM Compressioncited by 0
  19. 2025Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Modelscited by 0
  20. 2026Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Drivingcited by 0
  21. 2026ReAD: Reinforcement-Guided Capability Distillation for Large Language Modelscited by 0
  22. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0
  23. 2026ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generationcited by 0
  24. 2026AdaStop: Cost-Aware Early Stopping for DNN Test Selectioncited by 0
  25. 2026Reassembling Distributed Risk: Trajectory-Conditioned Action Generation for Multi-Turn Agent Safetycited by 0
  26. 2026FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Modelscited by 0
  27. 2026ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Modelscited by 0
  28. 2026A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensionscited by 0
  29. 2026Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillationcited by 0
  30. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models
  2. 2021Dynamic Knowledge Distillation for Pre-trained Language Models
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
  4. 2026Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
  5. 2025Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation
  6. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  7. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Models
  8. 2024Pre-training Distillation for Large Language Models: A Design Space Exploration