# Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models

arXiv: https://arxiv.org/abs/2404.02657 · HTML: https://mlsanity.com/paper/2404.02657/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints](https://mlsanity.com/paper/2309.16240/) (2023)
- [BiLLM: Pushing the Limit of Post-Training Quantization for LLMs](https://mlsanity.com/paper/2402.04291/) (2024)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2307.15190/) (2023)
- [Distilling Policy Distillation](https://mlsanity.com/paper/1902.02186/) (2019)
- [Compression of Generative Pre-trained Language Models via Quantization](https://mlsanity.com/paper/2203.10705/) (2022)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences](https://mlsanity.com/paper/2107.08285/) (2021)
- [KL Guided Domain Adaptation](https://mlsanity.com/paper/2106.07780/) (2021)
- [PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning](https://mlsanity.com/paper/2402.12842/) (2024)
- [Token-Scaled Logit Distillation for Ternary Weight Generative Language Models](https://mlsanity.com/paper/2308.06744/) (2023)
- [Self-Knowledge Distillation via Dropout](https://mlsanity.com/paper/2208.05642/) (2022)

## Built on by (most cited first, 30 of 65)

- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs](https://mlsanity.com/paper/2503.07067/) (2025)
- [LLaVA-KD: A Framework of Distilling Multimodal Large Language Models](https://mlsanity.com/paper/2410.16236/) (2024)
- [When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning](https://mlsanity.com/paper/2605.21606/) (2026)
- [MiniPLM: Knowledge Distillation for Pre-Training Language Models](https://mlsanity.com/paper/2410.17215/) (2024)
- [PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence](https://mlsanity.com/paper/2603.11178/) (2026)
- [Visual-Advantage On-Policy Distillation for Vision-Language Models](https://mlsanity.com/paper/2605.21924/) (2026)
- [ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence](https://mlsanity.com/paper/2505.04560/) (2025)
- [TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models](https://mlsanity.com/paper/2501.16937/) (2025)
- [SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs](https://mlsanity.com/paper/2510.24021/) (2025)
- [A Comedy of Estimators: On KL Regularization in RL Training of LLMs](https://mlsanity.com/paper/2512.21852/) (2025)
- [LLM-Oriented Token-Adaptive Knowledge Distillation](https://mlsanity.com/paper/2510.11615/) (2025)
- [SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation](https://mlsanity.com/paper/2605.07711/) (2026)
- [Shadow-FT: Tuning Instruct Model via Training on Paired Base Model](https://mlsanity.com/paper/2505.12716/) (2025)
- [GenRecal: Generation after Recalibration from Large to Small Vision-Language Models](https://mlsanity.com/paper/2506.15681/) (2025)
- [Multi-Rollout On-Policy Distillation via Peer Successes and Failures](https://mlsanity.com/paper/2605.12652/) (2026)
- [Hybrid Policy Distillation for LLMs](https://mlsanity.com/paper/2604.20244/) (2026)
- [A Dual-Space Framework for General Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2504.11426/) (2025)
- [Distilled Reinforcement Learning for LLM Post-training](https://mlsanity.com/paper/2607.17247/) (2026)
- [SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought](https://mlsanity.com/paper/2505.24181/) (2025)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Advantage-Guided Distillation for Preference Alignment in Small Language Models](https://mlsanity.com/paper/2502.17927/) (2025)
- [Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generator](https://mlsanity.com/paper/2503.15457/) (2025)
- [Distillation of Large Language Models via Concrete Score Matching](https://mlsanity.com/paper/2509.25837/) (2025)
- [Adversarial Moment-Matching Distillation of Large Language Models](https://mlsanity.com/paper/2406.02959/) (2024)
- [LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2411.06839/) (2024)
- [Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models](https://mlsanity.com/paper/2409.12512/) (2024)
- [GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training](https://mlsanity.com/paper/2512.13043/) (2025)
- [Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients](https://mlsanity.com/paper/2606.18216/) (2026)

## Abstract (arXiv)

Kullback-Leiber divergence has been widely used in Knowledge Distillation (KD) to compress Large Language Models (LLMs). Contrary to prior assertions that reverse Kullback-Leibler (RKL) divergence is mode-seeking and thus preferable over the mean-seeking forward Kullback-Leibler (FKL) divergence, this study empirically and theoretically demonstrates that neither mode-seeking nor mean-seeking properties manifest in KD for LLMs. Instead, RKL and FKL are found to share the same optimization objective and both converge after a sufficient number of epochs. However, due to practical constraints, LLMs are seldom trained for such an extensive number of epochs. Meanwhile, we further find that RKL focuses on the tail part of the distributions, while FKL focuses on the head part at the beginning epochs. Consequently, we propose a simple yet effective Adaptive Kullback-Leiber (AKL) divergence method, which adaptively allocates weights to combine FKL and RKL. Metric-based and GPT-4-based evaluations demonstrate that the proposed AKL outperforms the baselines across various tasks and improves the diversity and quality of generated responses. Codes are available at hrefhttps://github.com/wutaiqiang/LLMKDAKLgithub.