# Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again

arXiv: https://arxiv.org/abs/2210.04427 · HTML: https://mlsanity.com/paper/2210.04427/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 47 of its 68 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [LogME: Practical Assessment of Pre-trained Models for Transfer Learning](https://mlsanity.com/paper/2102.11005/) (2021)
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)

## Built on by (most cited first, 23 of 23)

- [Knowledge Diffusion for Distillation](https://mlsanity.com/paper/2305.15712/) (2023)
- [ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence](https://mlsanity.com/paper/2505.04560/) (2025)
- [Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation](https://mlsanity.com/paper/2306.16170/) (2023)
- [Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation](https://mlsanity.com/paper/2303.11611/) (2023)
- [Designing Large Foundation Models for Efficient Training and Inference: A Survey](https://mlsanity.com/paper/2409.01990/) (2024)
- [Dynamic Temperature Knowledge Distillation](https://mlsanity.com/paper/2404.12711/) (2024)
- [Cosine Similarity Knowledge Distillation for Individual Class Information Transfer](https://mlsanity.com/paper/2311.14307/) (2023)
- [Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning](https://mlsanity.com/paper/2311.13934/) (2023)
- [Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch](https://mlsanity.com/paper/2405.13078/) (2024)
- [Local Dense Logit Relations for Enhanced Knowledge Distillation](https://mlsanity.com/paper/2507.15911/) (2025)
- [Boosting Residual Networks with Group Knowledge](https://mlsanity.com/paper/2308.13772/) (2023)
- [Maximizing Discrimination Capability of Knowledge Distillation with Energy Function](https://mlsanity.com/paper/2311.14334/) (2023)
- [LoCa: Logit Calibration for Knowledge Distillation](https://mlsanity.com/paper/2409.04778/) (2024)
- [Lightweight Self-Knowledge Distillation with Multi-source Information Fusion](https://mlsanity.com/paper/2305.09183/) (2023)
- [Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks](https://mlsanity.com/paper/2405.12489/) (2024)
- [Low-Dimensional Federated Knowledge Graph Embedding via Knowledge Distillation](https://mlsanity.com/paper/2408.05748/) (2024)
- [Lightweight Contenders: Navigating Semi-Supervised Text Mining through Peer Collaboration and Self Transcendence](https://mlsanity.com/paper/2412.00883/) (2024)
- [MMARD: Improving the Min-Max Optimization Process in Adversarial Robustness Distillation](https://mlsanity.com/paper/2503.06559/) (2025)
- [iCD: A Implicit Clustering Distillation Mathod for Structural Information Mining](https://mlsanity.com/paper/2509.12553/) (2025)
- [Heterogeneous Complementary Distillation](https://mlsanity.com/paper/2511.10942/) (2025)
- [ADS-C: Antidistillation Sampling for Classification](https://mlsanity.com/paper/2607.15467/) (2026)
- [When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation](https://mlsanity.com/paper/2609.34489/) (2026)
- [Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models](https://mlsanity.com/paper/2609.36734/) (2026)

## Abstract (arXiv)

Knowledge Distillation (KD) aims at transferring the knowledge of a well-performed neural network (the it teacher) to a weaker one (the it student). A peculiar phenomenon is that a more accurate model doesn't necessarily teach better, and temperature adjustment can neither alleviate the mismatched capacity. To explain this, we decompose the efficacy of KD into three parts: it correct guidance, it smooth regularization, and it class discriminability. The last term describes the distinctness of it wrong class probabilities that the teacher provides in KD. Complex teachers tend to be over-confident and traditional temperature scaling limits the efficacy of it class discriminability, resulting in less discriminative wrong class probabilities. Therefore, we propose it Asymmetric Temperature Scaling (ATS), which separately applies a higher/lower temperature to the correct/wrong class. ATS enlarges the variance of wrong class probabilities in the teacher's label and makes the students grasp the absolute affinities of wrong classes to the target class as discriminative as possible. Both theoretical analysis and extensive experimental results demonstrate the effectiveness of ATS. The demo developed in Mindspore is available at https://gitee.com/lxcnju/ats-mindspore and will be available at https://gitee.com/mindspore/models/tree/master/research/cv/ats.