mlsanityData to 14 Oct 2026

On double-descent in uncertainty quantification in overparametrized models

arXiv 2210.12760 ↗First posted 23 Oct 2022cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Uncertainty quantification is a central challenge in reliable and trustworthy machine learning. Naive measures such as last-layer scores are well-known to yield overconfident estimates in the context of overparametrized neural networks. Several methods, ranging from temperature scaling to different Bayesian treatments of neural networks, have been proposed to mitigate overconfidence, most often supported by the numerical observation that they yield better calibrated uncertainty measures. In this work, we provide a sharp comparison between popular uncertainty measures for binary classification in a mathematically tractable model for overparametrized neural networks: the random features model. We discuss a trade-off between classification accuracy and calibration, unveiling a double descent like behavior in the calibration curve of optimally regularized estimators as a function of overparametrization. This is in contrast with the empirical Bayes method, which we show to be well calibrated in our setting despite the higher generalization error and overparametrization.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 52 of its 114 references that are in the map. First posted on arXiv 23 Oct 2022.

Builds on

52 of 114 references.

References · 52

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  5. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  6. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  7. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  9. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  10. 2019Benign Overfitting in Linear Regressioncited by 549
  11. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  12. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
38 more
  1. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  2. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  3. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  4. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 338
  5. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  6. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  7. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  8. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  9. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  10. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  11. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
  12. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  13. 2020Optimal Regularization Can Mitigate Double Descentcited by 109
  14. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  15. 2018A modern maximum-likelihood theory for high-dimensional logistic regressioncited by 81
  16. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  17. 2017A Random Matrix Approach to Neural Networkscited by 80
  18. 2022Universality of empirical risk minimizationcited by 75
  19. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 67
  20. 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 62
  21. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  22. 2018Entropy and mutual information in models of deep neural networkscited by 51
  23. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  24. 2018The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regressioncited by 46
  25. 2013A framework to characterize performance of LASSO algorithmscited by 40
  26. 2021Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classificationcited by 39
  27. 2021Graph-based Approximate Message Passing Iterationscited by 35
  28. 2020The estimation error of general first order methodscited by 33
  29. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  30. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 28
  31. 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
  32. 2020On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimationcited by 23
  33. 2019The spiked matrix model with generative priorscited by 20
  34. 2022Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimensioncited by 19
  35. 2022Theoretical characterization of uncertainty in high-dimensional linear classificationcited by 18
  36. 2022Learning curves for the multi-class teacher-student perceptroncited by 17
  37. 2021Structured Stochastic Gradient MCMCcited by 12
  38. 2020Strong replica symmetry for high-dimensional disordered log-concave Gibbs measurescited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 3; 2025: 4; 2026: 3.

16 papers.

Sort:

  1. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  2. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  3. 2023When Does Optimizing a Proper Loss Yield Calibration?cited by 36
  4. 2023High-dimensional robust regression under heavy-tailed data: Asymptotics and Universalitycited by 15
  5. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  6. 2024A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offscited by 7
  7. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 6
  8. 2024Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regressioncited by 4
  9. 2025Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structurecited by 4
  10. 2023Bayes optimal learning in high-dimensional linear regression with network side informationcited by 2
  11. 2024Building Conformal Prediction Intervals with Approximate Message Passingcited by 2
  12. 2025On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classificationcited by 2
4 more
  1. 2023Asymptotics of Bayesian Uncertainty Estimation in Random Features Regressioncited by 1
  2. 2026The Well-Tempered Classifier: Some Elementary Properties of Temperature Scalingcited by 1
  3. 2026Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descentcited by 0
  4. 2026LoRA's Second Descent Extends Beyond Parameter Paritycited by 0