mlsanityData to 7 Oct 2026

Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification

arXiv 2102.07856 ↗First posted 15 Feb 2021cs.LG math.ST stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

Modern machine learning models with high accuracy are often miscalibrated -- the predicted top probability does not reflect the actual accuracy, and tends to be over-confident. It is commonly believed that such over-confidence is mainly due to over-parametrization, in particular when the model is large enough to memorize the training data and maximize the confidence. In this paper, we show theoretically that over-parametrization is not the only reason for over-confidence. We prove that logistic regression is inherently over-confident, in the realizable, under-parametrized setting where the data is generated from the logistic model, and the sample size is much larger than the number of parameters. Further, this over-confidence happens for general well-specified binary classification problems as long as the activation is symmetric and concave on the positive part. Perhaps surprisingly, we also show that over-confidence is not always the case -- there exists another activation function (and a suitable loss function) under which the learned classifier is under-confident at some probability values. Overall, our theory provides a precise characterization of calibration in realizable binary classification, which we verify on simulations and real data experiments.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 29 of its 50 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

29 of 50 references.

References · 29

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2016Wide Residual Networkscited by 3,057
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  7. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  8. 2019Measuring Calibration in Deep Learningcited by 443
  9. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  10. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  11. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  12. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
17 more
  1. 2019Verified Uncertainty Calibrationcited by 307
  2. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  3. 2017Concrete Dropoutcited by 227
  4. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  5. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  6. 2019Ensemble Distribution Distillationcited by 158
  7. 2020Local Temperature Scaling for Probability Calibrationcited by 95
  8. 2018A modern maximum-likelihood theory for high-dimensional logistic regressioncited by 79
  9. 2020Distribution-free binary classification: prediction sets, confidence intervals and calibrationcited by 79
  10. 2020Intra Order-preserving Functions for Calibration of Multi-Class Neural Networkscited by 65
  11. 2020Moment Multicalibration for Uncertainty Estimationcited by 63
  12. 2018Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Controlcited by 54
  13. 2018The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regressioncited by 45
  14. 2013A framework to characterize performance of LASSO algorithmscited by 38
  15. 2020Hydra: Preserving Ensemble Diversity for Model Distillationcited by 31
  16. 2020Sample Complexity of Uniform Convergence for Multicalibrationcited by 24
  17. 2020Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensionscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 8; 2024: 11; 2025: 3; 2026: 5.

39 papers.

Sort:

By area:

  1. 2021Assessing Generalization of SGD via Disagreementcited by 116
  2. 2024Conformal Prediction Sets Improve Human Decision Makingcited by 35
  3. 2021Early-stopped neural networks are consistentcited by 34
  4. 2022Calibrated ensembles can mitigate accuracy tradeoffs under distribution shiftcited by 34
  5. 2023Beyond Confidence: Reliable Models Should Also Consider Atypicalitycited by 25
  6. 2022T-Cal: An optimal test for the calibration of predictive modelscited by 24
  7. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  8. 2022Theoretical characterization of uncertainty in high-dimensional linear classificationcited by 18
  9. 2023MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detectioncited by 17
  10. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 15
  11. 2023High-dimensional robust regression under heavy-tailed data: Asymptotics and Universalitycited by 15
  12. 2021Understanding the Under-Coverage Bias in Uncertainty Estimationcited by 10
27 more
  1. 2023Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Groupingcited by 10
  2. 2023Expectation consistency for calibration of neural networkscited by 9
  3. 2023Federated Inference with Reliable Uncertainty Quantification over Wireless Channels via Conformal Predictioncited by 9
  4. 2024Doubly Calibrated Estimator for Recommendation on Data Missing Not At Randomcited by 7
  5. 2024Selective Learning: Towards Robust Calibration with Dynamic Regularizationcited by 5
  6. 2022Calibrating for Class Weights by Modeling Machine Learningcited by 4
  7. 2024Confidence-Aware Multi-Field Model Calibrationcited by 4
  8. 2024Pearls from Pebbles: Improved Confidence Functions for Auto-labelingcited by 3
  9. 2025Trustworthy image-to-image translation: evaluating uncertainty calibration in unpaired training scenarioscited by 3
  10. 2021On Deep Neural Network Calibration by Regularization and its Impact on Refinementcited by 2
  11. 2022Heterogeneous Calibration: A post-hoc model-agnostic framework for improved generalizationcited by 2
  12. 2023Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendationcited by 2
  13. 2024Building Conformal Prediction Intervals with Approximate Message Passingcited by 2
  14. 2025Optimal and Provable Calibration in High-Dimensional Binary Classification: Angular Calibration and Platt Scalingcited by 2
  15. 2023DifAugGAN: A Practical Diffusion-style Data Augmentation for GAN-based Single Image Super-resolutioncited by 1
  16. 2024Decoupling Decision-Making in Fraud Prevention through Classifier Calibration for Business Logic Actioncited by 1
  17. 2024Safe reinforcement learning in uncertain contextscited by 1
  18. 2025Large Kernel MedNeXt for Breast Tumor Segmentation and Self-Normalizing Network for pCR Classification in Magnetic Resonance Imagescited by 1
  19. 2026The Well-Tempered Classifier: Some Elementary Properties of Temperature Scalingcited by 1
  20. 2026Confidence Calibration under Ambiguous Ground Truthcited by 1
  21. 2021Explaining generalization in deep learning: progress and fundamental limitscited by 0
  22. 2022Excess risk analysis for epistemic uncertainty with application to variational inferencecited by 0
  23. 2024From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibrationcited by 0
  24. 2024Uncertainty quantification for improving radiomic-based models in radiation pneumonitis predictioncited by 0
  25. 2026The Manokhin Probability Matrix: A Diagnostic Framework for Classifier Probability Qualitycited by 0
  26. 2026Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributionscited by 0
  27. 2026Uncertainty of Vision Medical Foundation Modelscited by 0