# Temperature-Adaptive Transformed Teacher Matching

arXiv: https://arxiv.org/abs/2608.29099 · HTML: https://mlsanity.com/paper/2608.29099/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 29 Aug 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022)
- [Logit Standardization in Knowledge Distillation](https://mlsanity.com/paper/2403.01427/) (2024)
- [Knowledge Distillation Based on Transformed Teacher Matching](https://mlsanity.com/paper/2402.11148/) (2024)
- [Conditional Mutual Information Constrained Deep Learning for Classification](https://mlsanity.com/paper/2309.09123/) (2023)
- [EA-KD: Entropy-based Adaptive Knowledge Distillation](https://mlsanity.com/paper/2311.13621/) (2023)
- [Towards Undistillable Models by Minimizing Conditional Mutual Information](https://mlsanity.com/paper/2507.00012/) (2025)
- [Flatness-aware Curriculum Learning via Adversarial Difficulty](https://mlsanity.com/paper/2508.18726/) (2025)

## Abstract (arXiv)

Temperature scaling is a core component of knowledge distillation, yet its role and effect are still not fully understood. Transformed Teacher Matching (TTM) clarifies the role of temperature scaling by applying it only to the teacher distribution and interpreting the resulting objective as standard distillation with an implicit R'enyi entropy regularization on the student. However, TTM still relies on a fixed temperature and does not specify how the teacher-side temperature should be adapted for individual samples. In this paper, we introduce a sample-wise inverse-temperature update for TTM by locally minimizing the Kullback-Leibler divergence between the temperature-scaled teacher distribution and the student's prediction. We derive closed-form first and second derivatives with respect to the inverse temperature, and show that they can be expressed using variance and covariance statistics of centered teacher and student logits under the transformed teacher weighting. This yields an efficient curvature-aware update that requires one softmax evaluation and a constant number of class-wise weighted sums. Experiments on standard image classification distillation benchmarks show that our temperature adaptation generally improves TTM and WTTM, while remaining competitive with or outperforming prior temperature-adaptive distillation baselines.