# The Interpolating Information Criterion for Overparameterized Models

arXiv: https://arxiv.org/abs/2307.07785 · HTML: https://mlsanity.com/paper/2307.07785/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 83 references that are in the map.
- First posted on arXiv 15 Jul 2023.

## Builds on (most cited first, 30 of 38)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017)
- [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel](https://mlsanity.com/paper/2010.15110/) (2020)
- [On the Origin of Implicit Regularization in Stochastic Gradient Descent](https://mlsanity.com/paper/2101.12176/) (2021)
- [A General Framework for Updating Belief Distributions](https://mlsanity.com/paper/1306.6430/) (2013)
- [Benign overfitting in ridge regression](https://mlsanity.com/paper/2009.14286/) (2020)
- [Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks](https://mlsanity.com/paper/2006.07322/) (2020)
- [Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks](https://mlsanity.com/paper/1904.13262/) (2019)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1709.01953/) (2017)
- [PAC-Bayesian Theory Meets Bayesian Inference](https://mlsanity.com/paper/1605.08636/) (2016)
- [Hessian-based Analysis of Large Batch Training and Robustness to Adversaries](https://mlsanity.com/paper/1802.08241/) (2018)
- [A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent](https://mlsanity.com/paper/2006.05013/) (2020)
- [Triple descent and the two kinds of overfitting: Where & why do they appear?](https://mlsanity.com/paper/2006.03509/) (2020)
- [Fast Finite Width Neural Tangent Kernel](https://mlsanity.com/paper/2206.08720/) (2022)
- [Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2002.06262/) (2020)
- [On the Maximum Hessian Eigenvalue and Generalization](https://mlsanity.com/paper/2206.10654/) (2022)
- [Deep Learning is Singular, and That's Good](https://mlsanity.com/paper/2010.11560/) (2020)
- [A Brief Prehistory of Double Descent](https://mlsanity.com/paper/2004.04328/) (2020)
- [SAM as an Optimal Relaxation of Bayes](https://mlsanity.com/paper/2210.01620/) (2022)

## Built on by (most cited first, 12 of 12)

- [AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality](https://mlsanity.com/paper/2410.10054/) (2024)
- [Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEs](https://mlsanity.com/paper/2403.10642/) (2024)
- [Uncertainty Quantification with the Empirical Neural Tangent Kernel](https://mlsanity.com/paper/2502.02870/) (2025)
- [Models of Heavy-Tailed Mechanistic Universality](https://mlsanity.com/paper/2506.03470/) (2025)
- [A PAC-Bayesian Perspective on the Interpolating Information Criterion](https://mlsanity.com/paper/2311.07013/) (2023)
- [Determinant Estimation under Memory Constraints and Neural Scaling Laws](https://mlsanity.com/paper/2503.04424/) (2025)
- [Uncertainty-Aware Diagnostics for Physics-Informed Machine Learning](https://mlsanity.com/paper/2510.26121/) (2025)
- [Is the Last Layer Sufficient for Uncertainty Quantification?](https://mlsanity.com/paper/2605.30741/) (2026)
- [Temperature Optimization for Bayesian Deep Learning](https://mlsanity.com/paper/2410.05757/) (2024)
- [Spectral Estimation with Free Decompression](https://mlsanity.com/paper/2506.11994/) (2025)
- [A Regularization-Sharpness Tradeoff for Linear Interpolators](https://mlsanity.com/paper/2602.12680/) (2026)
- [Why Heavy-Tailed Weight Spectra Predict Model Quality](https://mlsanity.com/paper/2610.13123/) (2026)

## Abstract (arXiv)

The problem of model selection is considered for the setting of interpolating estimators, where the number of model parameters exceeds the size of the dataset. Classical information criteria typically consider the large-data limit, penalizing model size. However, these criteria are not appropriate in modern settings where overparameterized models tend to perform well. For any overparameterized model, we show that there exists a dual underparameterized model that possesses the same marginal likelihood, thus establishing a form of Bayesian duality. This enables more classical methods to be used in the overparameterized setting, revealing the Interpolating Information Criterion, a measure of model quality that naturally incorporates the choice of prior into the model selection. Our new information criterion accounts for prior misspecification, geometric and spectral properties of the model, and is numerically consistent with known empirical and theoretical behavior in this regime.