mlsanityData to 14 Oct 2026

Divergence controls entropy in distillation

arXiv 2610.03529 ↗First posted 2 Oct 2026cs.LG cs.CLcited by 0

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

Distillation has become a core primitive of large language model training, but its properties are not yet well understood. We take an entropic perspective, studying how the entropy of the student depends on the data and the divergence that define the distillation objective. We prove that forward KL inflates the entropy of the student above that of the teacher. Since cross-entropy training is a special case, this yields an identity that we verify quantitatively in pretraining and supervised finetuning. Other divergences come with no such guarantee: reverse KL deflates entropy until the gap between student and teacher gets too large, and interpolating between the two changes entropy smoothly early in training but abruptly at convergence. The lower entropy of on-policy distillation comes from token-level reverse KL, not from on-policy sampling. The divergence therefore acts as an implicit entropy regularizer, whose role is clearest in self-distillation: as conditioning on privileged information deflates entropy, the divergence hyperparameters that work best are those that compensate for it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 48 of its 66 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

48 of 66 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Tail-Aware Top-k On-Policy Distillation12 Aug
  2. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training29 Jun
  3. 2026On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity24 Jun
  4. 2026Trust Region On-Policy Distillation31 May
  5. 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation16 May
  6. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  7. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes26 Mar
  8. 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation19 Mar
  9. 2026Reinforcement Learning via Self-Distillation28 Jan
  10. 2026Self-Distillation Enables Continual Learning27 Jan
  11. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 37

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  9. 2025Gemma 3 Technical Reportcited by 1,824
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
25 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  3. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  4. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  6. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  7. 2026MiMo-V2-Flash Technical Reportcited by 191
  8. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  9. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  10. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  11. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 105
  12. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 98
  13. 2017Variational Sequential Monte Carlocited by 95
  14. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  15. 2019Distilling Policy Distillationcited by 84
  16. 2025Diverse Preference Optimizationcited by 52
  17. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 51
  18. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 51
  19. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 37
  20. 2023When Does Optimizing a Proper Loss Yield Calibration?cited by 36
  21. 2023Scaling Laws for Associative Memoriescited by 35
  22. 2023The Shrinkage-Delinkage Trade-off: An Analysis of Factorized Gaussian Approximations for Variational Inferencecited by 10
  23. 2023Expectation consistency for calibration of neural networkscited by 10
  24. 2026Annotations Mitigate Post-Training Mode Collapsecited by 5
  25. 2026Language models suffer from a curse of ambiguitycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
  2. 2026Entropy-Aware On-Policy Distillation of Language Models
  3. 2026Divergence-Routed On-Policy Distillation for Language Models
  4. 2026Rethinking Reverse KL as Adaptive Entropy Distillation
  5. 2026On the Generalization of Knowledge Distillation: An Information-Theoretic View
  6. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  7. 2026Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
  8. 2026Self-Distillation with Verbal Regularization