mlsanityData to 7 Oct 2026

Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models

arXiv 2404.02657 ↗First posted 3 Apr 2024cs.CL cs.AIcited by 60

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Kullback-Leiber divergence has been widely used in Knowledge Distillation (KD) to compress Large Language Models (LLMs). Contrary to prior assertions that reverse Kullback-Leibler (RKL) divergence is mode-seeking and thus preferable over the mean-seeking forward Kullback-Leibler (FKL) divergence, this study empirically and theoretically demonstrates that neither mode-seeking nor mean-seeking properties manifest in KD for LLMs. Instead, RKL and FKL are found to share the same optimization objective and both converge after a sufficient number of epochs. However, due to practical constraints, LLMs are seldom trained for such an extensive number of epochs. Meanwhile, we further find that RKL focuses on the tail part of the distributions, while FKL focuses on the head part at the beginning epochs. Consequently, we propose a simple yet effective Adaptive Kullback-Leiber (AKL) divergence method, which adaptively allocates weights to combine FKL and RKL. Metric-based and GPT-4-based evaluations demonstrate that the proposed AKL outperforms the baselines across various tasks and improves the diversity and quality of generated responses. Codes are available at hrefhttps://github.com/wutaiqiang/LLMKDAKLgithub.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 49 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

34 of 49 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
22 more
  1. 2016Sequence-Level Knowledge Distillationcited by 761
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  3. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  4. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  5. 2023Scaling Data-Constrained Language Modelscited by 391
  6. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  7. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  8. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  9. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  10. 2019Distilling Policy Distillationcited by 81
  11. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  12. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  13. 2024A Survey on Transformer Compressioncited by 44
  14. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
  15. 2021KL Guided Domain Adaptationcited by 35
  16. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuningcited by 24
  17. 2023Token-Scaled Logit Distillation for Ternary Weight Generative Language Modelscited by 20
  18. 2022Self-Knowledge Distillation via Dropoutcited by 18
  19. 2023RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixercited by 13
  20. 2023Weight-Inherited Distillation for Task-Agnostic BERT Compressioncited by 8
  21. 2022BD-KD: Balancing the Divergences for Online Knowledge Distillationcited by 7
  22. 2022A Disability Lens towards Biases in GPT-3 Generated Open-Ended Languagescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 22; 2026: 32.

60 papers.

Sort:

By area:

  1. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  2. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  3. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  4. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  5. 2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 26
  6. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
  7. 2026Visual-Advantage On-Policy Distillation for Vision-Language Modelscited by 22
  8. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 21
  9. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 19
  10. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMscited by 18
  11. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  12. 2025LLM-Oriented Token-Adaptive Knowledge Distillationcited by 12
38 more
  1. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  2. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  3. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  4. 2025Shadow-FT: Tuning Instruct Model via Training on Paired Base Modelcited by 7
  5. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thoughtcited by 7
  6. 2026Hybrid Policy Distillation for LLMscited by 7
  7. 2026Distilled Reinforcement Learning for LLM Post-trainingcited by 7
  8. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  9. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generatorcited by 5
  10. 2025A Dual-Space Framework for General Knowledge Distillation of Large Language Modelscited by 5
  11. 2024Adversarial Moment-Matching Distillation of Large Language Modelscited by 4
  12. 2024LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Modelscited by 4
  13. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 4
  14. 2025Distillation of Large Language Models via Concrete Score Matchingcited by 4
  15. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Modelscited by 3
  16. 2025GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Trainingcited by 3
  17. 2024LoCa: Logit Calibration for Knowledge Distillationcited by 2
  18. 2025HaLoRA: Hardware-aware Low-Rank Adaptation for Large Language Models Based on Hybrid Compute-in-Memory Architecturecited by 2
  19. 2025AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distributioncited by 2
  20. 2026EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillationcited by 2
  21. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillationcited by 2
  22. 2025Head-Tail-Aware KL Divergence in Knowledge Distillation for Spiking Neural Networkscited by 1
  23. 2025Frugal Machine Learning for Energy-efficient, and Resource-aware Artificial Intelligencecited by 1
  24. 2025Lifelong Learning with Behavior Consolidation for Vehicle Routingcited by 1
  25. 2026Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatchcited by 1
  26. 2026Consistently Informative Soft-Label Temperature for Knowledge Distillationcited by 1
  27. 2026DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillationcited by 1
  28. 2026RL Starts before RL: On Policy Distillation for Better Reinforcement Learningcited by 1
  29. 2025Do LLMs Align with My Task? Evaluating Text-to-SQL via Dataset Alignmentcited by 0
  30. 2025Winning the Lottery by Preserving Network Training Dynamics with Concrete Ticket Searchcited by 0
  31. 2026STAR: Similarity-guided Teacher-Assisted Refinement for Super-Tiny Function Calling Modelscited by 0
  32. 2026DistillLens: Symmetric Knowledge Distillation Through Logit Lenscited by 0
  33. 2026HEART-PFL: Stable Personalized Federated Learning under Heterogeneity with Hierarchical Directional Alignment and Adversarial Knowledge Transfercited by 0
  34. 2026Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Modelscited by 0
  35. 2026Distribution Corrected Offline Data Distillation for Large Language Modelscited by 0
  36. 2026DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillationcited by 0
  37. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0
  38. 2026Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillationcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking the Role of Temperature in Large Language Model Distillation
  2. 2025Better Estimation of the Kullback--Leibler Divergence Between Language Models
  3. 2024Direct Preference Knowledge Distillation for Large Language Models
  4. 2026WASD: Wasserstein-based Knowledge Distillation for Large Language Models
  5. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  6. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  7. 2024Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment
  8. 2020Autoregressive Knowledge Distillation through Imitation Learning