# A Unified Revisit of Temperature in Classification-Based Knowledge Distillation

arXiv: https://arxiv.org/abs/2603.02430 · HTML: https://mlsanity.com/paper/2603.02430/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 38 of its 57 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [Separable Self-attention for Mobile Vision Transformers](https://mlsanity.com/paper/2206.02680/) (2022)
- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022)
- [Logit Standardization in Knowledge Distillation](https://mlsanity.com/paper/2403.01427/) (2024)
- [MobileNetV4 -- Universal Models for the Mobile Ecosystem](https://mlsanity.com/paper/2404.10518/) (2024)
- [MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks](https://mlsanity.com/paper/2009.08453/) (2020)

## Abstract (arXiv)

A central idea of knowledge distillation is to expose relational structure embedded in the teacher's weights for the student to learn, which is often facilitated using a temperature parameter. Despite its widespread use, there remains limited understanding on how to select an appropriate temperature value, or how this value depends on other training elements such as optimizer, teacher pretraining/finetuning, etc. In practice, temperature is commonly chosen via grid search or by adopting values from prior work, which can be time-consuming or may lead to suboptimal student performance when training setups differ. In this work, we posit that temperature is closely linked to these training components and present a unified study that systematically examines such interactions. From analyzing these cross-connections, we identify and present common situations that have a pronounced impact on temperature selection, providing valuable guidance for practitioners employing knowledge distillation in their work.