mlsanityData to 7 Oct 2026

Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes

arXiv 2210.07612 ↗First posted 14 Oct 2022stat.ML cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Despite their importance for assessing reliability of predictions, uncertainty quantification (UQ) measures for machine learning models have only recently begun to be rigorously characterized. One prominent issue is the curse of dimensionality: it is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyperparameters to maximize marginal likelihood (the empirical Bayes procedure), the performance, as measured by the marginal likelihood, improves monotonically} with the input dimension. On the other hand, we prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent. Cold posteriors, which have recently attracted interest due to their improved performance in certain settings, appear to exacerbate these phenomena. We verify empirically that our results hold for real data, beyond our considered assumptions, and we explore consequences involving synthetic covariates.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 57 of its 94 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

57 of 94 references.

References · 57

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2018A Tutorial on Bayesian Optimizationcited by 734
  4. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  5. 2019Benign Overfitting in Linear Regressioncited by 544
  6. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  7. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  8. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  9. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  10. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  11. 2018To understand deep learning we need to understand kernel learningcited by 186
  12. 2020Benign overfitting in ridge regressioncited by 152
38 more
  1. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  2. 2020Bayesian Neural Networks: An Introduction and Surveycited by 130
  3. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  4. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  5. 2019Harmless interpolation of noisy data in regressioncited by 121
  6. 2021Bayesian Neural Network Priors Revisitedcited by 121
  7. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  8. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 104
  9. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  10. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  11. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  12. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  13. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  14. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  15. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  16. 2021A survey on high-dimensional Gaussian process modeling with application to Bayesian optimizationcited by 76
  17. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  18. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
  19. 2021The Shape of Learning Curves: a Reviewcited by 67
  20. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 65
  21. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 64
  22. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  23. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  24. 2020A statistical theory of cold posteriors in deep neural networkscited by 53
  25. 2020A Brief Prehistory of Double Descentcited by 44
  26. 2019Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networkscited by 43
  27. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
  28. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  29. 2019On the marginal likelihood and cross-validationcited by 36
  30. 2019Exact expressions for double descent and implicit regularization via surrogate random designcited by 35
  31. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  32. 2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27
  33. 2017Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behaviorcited by 24
  34. 2019Bayesian Optimization using Deep Gaussian Processescited by 22
  35. 2020Cold Posteriors and Aleatoric Uncertaintycited by 22
  36. 2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
  37. 2020Modern Monte Carlo Methods for Efficient Uncertainty Quantification and Propagation: A Surveycited by 21
  38. 2020Sparse sketches with small inversion biascited by 19

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 4; 2026: 2.

10 papers.

  1. 2024Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEscited by 14
  2. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 11
  3. 2023Expectation consistency for calibration of neural networkscited by 9
  4. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  5. 2025Determinant Estimation under Memory Constraints and Neural Scaling Lawscited by 3
  6. 2025Uncertainty-Aware Diagnostics for Physics-Informed Machine Learningcited by 3
  7. 2026Is the Last Layer Sufficient for Uncertainty Quantification?cited by 1
  8. 2025Spectral Estimation with Free Decompressioncited by 0
  9. 2025A Ridge Too Far: Correcting Over-Shrinkage via Negative Regularizationcited by 0
  10. 2026FluidGaussian: Propagating Simulation-Based Uncertainty Toward Functionally-Intelligent 3D Reconstructioncited by 0