mlsanityData to 14 Oct 2026

Layer-Stack Temperature Scaling

arXiv 2211.10193 ↗First posted 18 Nov 2022cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Recent works demonstrate that early layers in a neural network contain useful information for prediction. Inspired by this, we show that extending temperature scaling across all layers improves both calibration and accuracy. We call this procedure "layer-stack temperature scaling" (LATES). Informally, LATES grants each layer a weighted vote during inference. We evaluate it on five popular convolutional neural network architectures both in- and out-of-distribution and observe a consistent improvement over temperature scaling in terms of accuracy, calibration, and AUC. All conclusions are supported by comprehensive statistical analyses. Since LATES neither retrains the architecture nor introduces many more parameters, its advantages can be reaped without requiring additional data beyond what is used in temperature scaling. Finally, we show that combining LATES with Monte Carlo Dropout matches state-of-the-art results on CIFAR10/100.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 18 Nov 2022.

Builds on

37 of 55 references.

References · 37

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2016Wide Residual Networkscited by 3,070
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  11. 2014How transferable are features in deep neural networks?cited by 1,225
  12. 2017A Closer Look at Memorization in Deep Networkscited by 942
25 more
  1. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  2. 2017Selective Classification for Deep Neural Networkscited by 852
  3. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  4. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  5. 2019Measuring Calibration in Deep Learningcited by 446
  6. 2017Algorithmic decision making and the cost of fairnesscited by 403
  7. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  8. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 368
  9. 2014Deeply-Supervised Netscited by 367
  10. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  11. 2019Verified Uncertainty Calibrationcited by 309
  12. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  13. 2020Training independent subnetworks for robust predictioncited by 177
  14. 2018Classification with Fairness Constraints: A Meta-Algorithm with Provable Guaranteescited by 146
  15. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  16. 2020Calibration of Neural Networks using Splinescited by 97
  17. 2021Soft Calibration Objectives for Neural Networkscited by 92
  18. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  19. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 82
  20. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 68
  21. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 59
  22. 2019Non-Parametric Calibration for Classificationcited by 57
  23. 2020Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binningcited by 27
  24. 2021A Near-Optimal Algorithm for Debiasing Trained Machine Learning Modelscited by 19
  25. 2018DNN or k-NN: That is the Generalize vs. Memorize Questioncited by 16

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  2. 2024Post-Hoc Reversal: Are We Selecting Models Prematurely?cited by 8