mlsanityData to 14 Oct 2026

The Interpolating Information Criterion for Overparameterized Models

arXiv 2307.07785 ↗First posted 15 Jul 2023stat.ML cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

The problem of model selection is considered for the setting of interpolating estimators, where the number of model parameters exceeds the size of the dataset. Classical information criteria typically consider the large-data limit, penalizing model size. However, these criteria are not appropriate in modern settings where overparameterized models tend to perform well. For any overparameterized model, we show that there exists a dual underparameterized model that possesses the same marginal likelihood, thus establishing a form of Bayesian duality. This enables more classical methods to be used in the overparameterized setting, revealing the Interpolating Information Criterion, a measure of model quality that naturally incorporates the choice of prior into the model selection. Our new information criterion accounts for prior misspecification, geometric and spectral properties of the model, and is numerically consistent with known empirical and theoretical behavior in this regime.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 38 of its 83 references that are in the map. First posted on arXiv 15 Jul 2023.

Builds on

38 of 83 references.

References · 38

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  4. 2019Benign Overfitting in Linear Regressioncited by 549
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  6. 2017Implicit Regularization in Matrix Factorizationcited by 322
  7. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  8. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  9. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  10. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  11. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  12. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
26 more
  1. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  2. 2013A General Framework for Updating Belief Distributionscited by 175
  3. 2020Benign overfitting in ridge regressioncited by 153
  4. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 138
  5. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  6. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
  7. 2021The Bayesian Learning Rulecited by 99
  8. 2017Implicit Regularization in Deep Learningcited by 90
  9. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 88
  10. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  11. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 70
  12. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 65
  13. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  14. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  15. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  16. 2020Deep Learning is Singular, and That's Goodcited by 49
  17. 2020A Brief Prehistory of Double Descentcited by 44
  18. 2022SAM as an Optimal Relaxation of Bayescited by 41
  19. 2019On the marginal likelihood and cross-validationcited by 37
  20. 2019Exact expressions for double descent and implicit regularization via surrogate random designcited by 35
  21. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  22. 2013A Bayesian information criterion for singular modelscited by 22
  23. 2018Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learningcited by 20
  24. 2023Riemannian Laplace approximations for Bayesian neural networkscited by 15
  25. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 11
  26. 2021On quantitative Laplace-type convergence results for some exponential probability measures, with two applicationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 5; 2026: 3.

12 papers.

  1. 2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Qualitycited by 20
  2. 2024Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEscited by 15
  3. 2025Uncertainty Quantification with the Empirical Neural Tangent Kernelcited by 11
  4. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 10
  5. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 5
  6. 2025Determinant Estimation under Memory Constraints and Neural Scaling Lawscited by 4
  7. 2025Uncertainty-Aware Diagnostics for Physics-Informed Machine Learningcited by 3
  8. 2026Is the Last Layer Sufficient for Uncertainty Quantification?cited by 2
  9. 2024Temperature Optimization for Bayesian Deep Learningcited by 1
  10. 2025Spectral Estimation with Free Decompressioncited by 1
  11. 2026A Regularization-Sharpness Tradeoff for Linear Interpolatorscited by 0
  12. 2026Why Heavy-Tailed Weight Spectra Predict Model Qualitycited by 0