# Exploring the Impact of Temperature Scaling in Softmax for Classification and Adversarial Robustness

arXiv: https://arxiv.org/abs/2502.20604 · HTML: https://mlsanity.com/paper/2502.20604/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack](https://mlsanity.com/paper/1907.02044/) (2019)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networks](https://mlsanity.com/paper/2208.03160/) (2022)
- [Evaluating and Understanding the Robustness of Adversarial Logit Pairing](https://mlsanity.com/paper/1807.10272/) (2018)
- [Temperature check: theory and practice for training models with softmax-cross-entropy losses](https://mlsanity.com/paper/2010.07344/) (2020)
- [Label Smoothing and Logit Squeezing: A Replacement for Adversarial Training?](https://mlsanity.com/paper/1910.11585/) (2019)
- [Improving Adversarial Robustness with Self-Paced Hard-Class Pair Reweighting](https://mlsanity.com/paper/2210.15068/) (2022)
- [Constraining Logits by Bounded Function for Adversarial Robustness](https://mlsanity.com/paper/2010.02558/) (2020)

## Built on by (most cited first, 9 of 9)

- [Subquadratic Algorithms and Hardness for Attention with Any Temperature](https://mlsanity.com/paper/2505.14840/) (2025)
- [When Flatness Does (Not) Guarantee Adversarial Robustness](https://mlsanity.com/paper/2510.14231/) (2025)
- [GenoArmory: A Unified Evaluation Framework for Adversarial Attacks on Genomic Foundation Models](https://mlsanity.com/paper/2505.10983/) (2025)
- [Composable Score-based Graph Diffusion Model for Multi-Conditional Molecular Generation](https://mlsanity.com/paper/2509.09451/) (2025)
- [Demystifying When Pruning Works via Representation Hierarchies](https://mlsanity.com/paper/2603.24652/) (2026)
- [Sinkhorn doubly stochastic attention rank decay analysis](https://mlsanity.com/paper/2604.07925/) (2026)
- [Analytical Softmax Temperature Setting from Feature Dimensions for Model- and Domain-Robust Classification](https://mlsanity.com/paper/2504.15594/) (2025)
- [Deep Learning-based Binary Analysis for Vulnerability Detection in x86-64 Machine Code](https://mlsanity.com/paper/2601.09157/) (2026)
- [Toward Epistemic Stability: Engineering Consistent Procedures for Industrial LLM Hallucination Reduction](https://mlsanity.com/paper/2603.10047/) (2026)

## Abstract (arXiv)

The softmax function is a fundamental component in deep learning. This study delves into the often-overlooked parameter within the softmax function, known as "temperature," providing novel insights into the practical and theoretical aspects of temperature scaling for image classification. Our empirical studies, adopting convolutional neural networks and transformers on multiple benchmark datasets, reveal that moderate temperatures generally introduce better overall performance. Through extensive experiments and rigorous theoretical analysis, we explore the role of temperature scaling in model training and unveil that temperature not only influences learning step size but also shapes the model's optimization direction. Moreover, for the first time, we discover a surprising benefit of elevated temperatures: enhanced model robustness against common corruption, natural perturbation, and non-targeted adversarial attacks like Projected Gradient Descent. We extend our discoveries to adversarial training, demonstrating that, compared to the standard softmax function with the default temperature value, higher temperatures have the potential to enhance adversarial training. The insights of this work open new avenues for improving model performance and security in deep learning applications.