mlsanityData to 7 Oct 2026

Revisiting the Calibration of Modern Neural Networks

arXiv 2106.07998 ↗First posted 15 Jun 2021cs.LG cs.CVcited by 387

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.

Key facts. Cited by 387 papers in the map (data through 7 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

39 of 57 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
27 more
  1. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  2. 2019Natural Adversarial Examplescited by 1,401
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  4. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  5. 2016End to End Learning for Self-Driving Carscited by 1,000
  6. 2019When Does Label Smoothing Help?cited by 985
  7. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  8. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  9. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  10. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  11. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  12. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  13. 2019Measuring Calibration in Deep Learningcited by 443
  14. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  15. 2019Verified Uncertainty Calibrationcited by 307
  16. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  17. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  18. 2020Training independent subnetworks for robust predictioncited by 177
  19. 2019Evaluating model calibration in classificationcited by 165
  20. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  21. 2020Uncertainty Quantification and Deep Ensemblescited by 138
  22. 2020MetNet: A Neural Weather Model for Precipitation Forecastingcited by 128
  23. 2020Mitigating Bias in Calibration Error Estimationcited by 100
  24. 2020Calibration of Neural Networks using Splinescited by 97
  25. 2020Combining Ensembles and Data Augmentation can Harm your Calibrationcited by 44
  26. 2021Supervised Transfer Learning at Scale for Medical Imagingcited by 38
  27. 2020Temporal Probability Calibrationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 63; 2023: 84; 2024: 84; 2025: 70; 2026: 75.

387 papers.

Sort:

By area:

  1. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  4. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  5. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  6. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 351
  7. 2022How Do Vision Transformers Work?cited by 305
  8. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  9. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  10. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  11. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  12. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
38 more
  1. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  2. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  3. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  4. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 82
  5. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  6. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  7. 2023On the Calibration of Large Language Models and Alignmentcited by 72
  8. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  9. 2022RbA: Segmenting Unknown Regions Rejected by Allcited by 67
  10. 2022Better Uncertainty Calibration via Proper Scores for Classification and Beyondcited by 65
  11. 2022Forecasting Future World Events with Neural Networkscited by 63
  12. 2022Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysiscited by 62
  13. 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 59
  14. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
  15. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  16. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  17. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  18. 2022A Unifying Theory of Distance from Calibrationcited by 51
  19. 2022A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibrationcited by 49
  20. 2024Calibration and Correctness of Language Models for Codecited by 49
  21. 2024A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)cited by 44
  22. 2023Rethinking Confidence Calibration for Failure Predictioncited by 43
  23. 2023Prompting is not a substitute for probability measurements in large language modelscited by 43
  24. 2024When to Trust LLMs: Aligning Confidence with Response Qualitycited by 42
  25. 2022Robust Calibration with Multi-domain Temperature Scalingcited by 39
  26. 2022Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Setscited by 38
  27. 2023Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Modelscited by 37
  28. 2023When Does Optimizing a Proper Loss Yield Calibration?cited by 35
  29. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  30. 2024Conformal Prediction Sets Improve Human Decision Makingcited by 35
  31. 2022An Impartial Take to the CNN vs Transformer Robustness Contestcited by 34
  32. 2022Predictive Inference with Feature Conformal Predictioncited by 33
  33. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  34. 2022Beyond calibration: estimating the grouping loss of modern neural networkscited by 32
  35. 2024Calibrating Long-form Generations from Large Language Modelscited by 31
  36. 2022Calibrated Selective Classificationcited by 30
  37. 2021Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Errorcited by 28
  38. 2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28

50 most cited of 387 All 387 (JSON)