# Rethinking the Role of Temperature in Large Language Model Distillation

arXiv: https://arxiv.org/abs/2606.00306 · HTML: https://mlsanity.com/paper/2606.00306/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 29 May 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 7 of 7)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Understanding and Improving Knowledge Distillation](https://mlsanity.com/paper/2002.03532/) (2020)
- [Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation](https://mlsanity.com/paper/2604.00223/) (2026)
- [Consistently Informative Soft-Label Temperature for Knowledge Distillation](https://mlsanity.com/paper/2605.20357/) (2026)

## Abstract (arXiv)

Reverse Kullback-Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation, yet this preference is largely based on comparisons that omit the temperature τ, overlooking its central role in softening teacher distributions and improving knowledge transfer. In this work, we revisit temperature in LLM distillation and show that it fundamentally changes the comparison between FKL and RKL. Our analysis reveals an asymmetric effect: temperature substantially enriches FKL with non-dominant token signals, whereas it mainly rescales RKL gradients, causing FKL to benefit much more from τ scaling than RKL. This asymmetry overturns the standard empirical conclusion: although RKL outperforms FKL at τ=1, FKL consistently surpasses RKL at higher temperatures across instruction-following benchmarks. Moreover, the impact of temperature is not limited to FKL; it improves a broader family of distillation objectives, enabling simple KL-based methods to achieve competitive performance against recent state-of-the-art LLM distillation approaches.