mlsanityData to 14 Oct 2026

A Benchmark Study on Calibration

arXiv 2308.11838 ↗First posted 23 Aug 2023cs.LG cs.AI stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

Deep neural networks are increasingly utilized in various machine learning tasks. However, as these models grow in complexity, they often face calibration issues, despite enhanced prediction accuracy. Many studies have endeavored to improve calibration performance through the use of specific loss functions, data preprocessing and training frameworks. Yet, investigations into calibration properties have been somewhat overlooked. Our study leverages the Neural Architecture Search (NAS) search space, offering an exhaustive model architecture space for thorough calibration properties exploration. We specifically create a model calibration dataset. This dataset evaluates 90 bin-based and 12 additional calibration measurements across 117,702 unique neural networks within the widely employed NATS-Bench search space. Our analysis aims to answer several longstanding questions in the field, using our proposed dataset: (i) Can model calibration be generalized across different datasets? (ii) Can robustness be used as a calibration measurement? (iii) How reliable are calibration metrics? (iv) Does a post-hoc calibration method affect all models uniformly? (v) How does calibration interact with accuracy? (vi) What is the impact of bin size on calibration measurement? (vii) Which architectural designs are beneficial for calibration? Additionally, our study bridges an existing gap by exploring calibration within NAS. By providing this dataset, we enable further research into NAS calibration. As far as we are aware, our research represents the first large-scale investigation into calibration properties and the premier study of calibration issues within NAS. The project page can be found at https://www.taolinwei.com/calibration-study

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 53 of its 67 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

53 of 67 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2016Wide Residual Networkscited by 3,070
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  12. 2021Vision Transformers for Dense Predictioncited by 1,803
38 more
  1. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  4. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  5. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  6. 2019When Does Label Smoothing Help?cited by 988
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  8. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  9. 2021Going deeper with Image Transformerscited by 764
  10. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  11. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  12. 2019Measuring Calibration in Deep Learningcited by 446
  13. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  14. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  15. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  16. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  17. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  18. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
  19. 2019Verified Uncertainty Calibrationcited by 309
  20. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  21. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 260
  22. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 254
  23. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 210
  24. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  25. 2019Evaluating model calibration in classificationcited by 168
  26. 2020Improving model calibration with accuracy versus uncertainty optimizationcited by 123
  27. 2020Self-Knowledge Distillation with Progressive Refinement of Targetscited by 107
  28. 2020Mitigating Bias in Calibration Error Estimationcited by 101
  29. 2020NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Sizecited by 98
  30. 2020Calibration of Neural Networks using Splinescited by 97
  31. 2021Soft Calibration Objectives for Neural Networkscited by 92
  32. 2019One-Shot Neural Architecture Search via Self-Evaluated Template Networkcited by 69
  33. 2021When and How Mixup Improves Calibrationcited by 60
  34. 2021Distribution-free calibration guarantees for histogram binning without sample splittingcited by 45
  35. 2023Dual Focal Loss for Calibrationcited by 33
  36. 2019Can We Trust You? On Calibration of a Probabilistic Object Detector for Autonomous Drivingcited by 30
  37. 2020Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binningcited by 27
  38. 2020Exploring the Loss Landscape in Neural Architecture Searchcited by 20

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 3; 2026: 5.

11 papers.

  1. 2023Forking Uncertainties: Reliable Prediction and Model Predictive Control with Sequence Models via Conformal Risk Controlcited by 20
  2. 2025Rethinking Early Stopping: Refine, Then Calibratecited by 17
  3. 2024Calibrating Bayesian Learning via Regularization, Confidence Minimization, and Selective Inferencecited by 7
  4. 2025Uncertainty Weighted Gradients for Model Calibrationcited by 5
  5. 2024Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighborscited by 3
  6. 2026Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMscited by 3
  7. 2025Beyond One-Hot Labels: Semantic Mixing for Model Calibrationcited by 2
  8. 2026CalArena: A Large-Scale Post-Hoc Calibration Benchmarkcited by 2
  9. 2026Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shiftscited by 1
  10. 2026Beyond Output-Space Calibration: Spectral Evidence Bundling for Selective Reliability Estimation in Time-Series Classificationcited by 0
  11. 2026Calibration in One Go: Jointly Learning Accurate and Calibrated Neural Networkscited by 0