mlsanityData to 7 Oct 2026

Rethinking the Role of Temperature in Large Language Model Distillation

arXiv 2606.00306 ↗First posted 29 May 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Reverse Kullback-Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation, yet this preference is largely based on comparisons that omit the temperature τ, overlooking its central role in softening teacher distributions and improving knowledge transfer. In this work, we revisit temperature in LLM distillation and show that it fundamentally changes the comparison between FKL and RKL. Our analysis reveals an asymmetric effect: temperature substantially enriches FKL with non-dominant token signals, whereas it mainly rescales RKL gradients, causing FKL to benefit much more from τ scaling than RKL. This asymmetry overturns the standard empirical conclusion: although RKL outperforms FKL at τ=1, FKL consistently surpasses RKL at higher temperatures across instruction-following benchmarks. Moreover, the impact of temperature is not limited to FKL; it improves a broader family of distillation objectives, enabling simple KL-based methods to achieve competitive performance against recent state-of-the-art LLM distillation approaches.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 7 of its 33 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

7 of 33 references.

References · 7

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  5. 2020Understanding and Improving Knowledge Distillationcited by 83
  6. 2026Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillationcited by 5
  7. 2026Consistently Informative Soft-Label Temperature for Knowledge Distillationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
  2. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  3. 2026Rethinking Reverse KL as Adaptive Entropy Distillation
  4. 2025Optimizing Temperature for Language Models with Multi-Sample Inference
  5. 2026WASD: Wasserstein-based Knowledge Distillation for Large Language Models
  6. 2024Direct Preference Knowledge Distillation for Large Language Models
  7. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
  8. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models