# Temperature check: theory and practice for training models with softmax-cross-entropy losses

arXiv: https://arxiv.org/abs/2010.07344 · HTML: https://mlsanity.com/paper/2010.07344/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2020.

## Built on by (most cited first, 30 of 31)

- [Temperature as Uncertainty in Contrastive Learning](https://mlsanity.com/paper/2110.04403/) (2021)
- [SAM operates far from home: eigenvalue regularization as a dynamical phenomenon](https://mlsanity.com/paper/2302.08692/) (2023)
- [Training Transformers with Enforced Lipschitz Constants](https://mlsanity.com/paper/2507.13338/) (2025)
- [The Optimization Landscape of SGD Across the Feature Learning Strength](https://mlsanity.com/paper/2410.04642/) (2024)
- [Neglected Hessian component explains mysteries in Sharpness regularization](https://mlsanity.com/paper/2401.10809/) (2024)
- [Feature learning as alignment: a structural property of gradient descent in non-linear neural networks](https://mlsanity.com/paper/2402.05271/) (2024)
- [Exploring the Impact of Temperature Scaling in Softmax for Classification and Adversarial Robustness](https://mlsanity.com/paper/2502.20604/) (2025)
- [Tilting the playing field: Dynamical loss functions for machine learning](https://mlsanity.com/paper/2102.03793/) (2021)
- [MAP: Model Aggregation and Personalization in Federated Learning with Incomplete Classes](https://mlsanity.com/paper/2404.09232/) (2024)
- [SimSC: A Simple Framework for Semantic Correspondence with Temperature Learning](https://mlsanity.com/paper/2305.02385/) (2023)
- [LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2411.00918/) (2024)
- [LILogic Net: Compact Logic Gate Networks with Learnable Connectivity for Efficient Hardware Deployment](https://mlsanity.com/paper/2511.12340/) (2025)
- [Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty](https://mlsanity.com/paper/2209.09658/) (2022)
- [Improving Local Training in Federated Learning via Temperature Scaling](https://mlsanity.com/paper/2401.09986/) (2024)
- [Subquadratic Algorithms and Hardness for Attention with Any Temperature](https://mlsanity.com/paper/2505.14840/) (2025)
- [Deconstructing the Goldilocks Zone of Neural Network Initialization](https://mlsanity.com/paper/2402.03579/) (2024)
- [Extreme Miscalibration and the Illusion of Adversarial Robustness](https://mlsanity.com/paper/2402.17509/) (2024)
- [Grassmannian learning mutual subspace method for image set recognition](https://mlsanity.com/paper/2111.04352/) (2021)
- [Learning to Scale Temperature in Masked Self-Attention for Image Inpainting](https://mlsanity.com/paper/2302.06130/) (2023)
- [Explaining Grokking in Transformers through the Lens of Inductive Bias](https://mlsanity.com/paper/2602.06702/) (2026)
- [Sinkhorn doubly stochastic attention rank decay analysis](https://mlsanity.com/paper/2604.07925/) (2026)
- [Temperature Optimization for Bayesian Deep Learning](https://mlsanity.com/paper/2410.05757/) (2024)
- [A Novel Streamline-based diffusion MRI Tractography Registration Method with Probabilistic Keypoint Detection](https://mlsanity.com/paper/2503.02481/) (2025)
- [Probability-Dependent Gradient Decay in Large Margin Softmax](https://mlsanity.com/paper/2210.17145/) (2022)
- [Analytical Softmax Temperature Setting from Feature Dimensions for Model- and Domain-Robust Classification](https://mlsanity.com/paper/2504.15594/) (2025)
- [Comparative sentiment analysis of public perception: Monkeypox vs. COVID-19 behavioral insights](https://mlsanity.com/paper/2505.07430/) (2025)
- [Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalization](https://mlsanity.com/paper/2602.00827/) (2026)
- [Temperature Scaling Attack Disrupting Model Confidence in Federated Learning](https://mlsanity.com/paper/2602.06638/) (2026)
- [Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy](https://mlsanity.com/paper/2603.13552/) (2026)
- [Scalable, Energy-Efficient Optical-Neural Architecture for Multiplexed Deepfake Video Detection](https://mlsanity.com/paper/2605.19360/) (2026)

## Abstract (arXiv)

The softmax function combined with a cross-entropy loss is a principled approach to modeling probability distributions that has become ubiquitous in deep learning. The softmax function is defined by a lone hyperparameter, the temperature, that is commonly set to one or regarded as a way to tune model confidence after training; however, less is known about how the temperature impacts training dynamics or generalization performance. In this work we develop a theory of early learning for models trained with softmax-cross-entropy loss and show that the learning dynamics depend crucially on the inverse-temperature β as well as the magnitude of the logits at initialization, ||βbf z||2. We follow up these analytic results with a large-scale empirical study of a variety of model architectures trained on CIFAR10, ImageNet, and IMDB sentiment analysis. We find that generalization performance depends strongly on the temperature, but only weakly on the initial logit magnitude. We provide evidence that the dependence of generalization on β is not due to changes in model confidence, but is a dynamical phenomenon. It follows that the addition of β as a tunable hyperparameter is key to maximizing model performance. Although we find the optimal β to be sensitive to the architecture, our results suggest that tuning β over the range 10-2 to 101 improves performance over all architectures studied. We find that smaller β may lead to better peak performance at the cost of learning stability.