# Bayesian Analysis for Over-parameterized Linear Model via Effective Spectra

arXiv: https://arxiv.org/abs/2305.15754 · HTML: https://mlsanity.com/paper/2305.15754/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 66 references that are in the map.
- First posted on arXiv 25 May 2023.

## Builds on (most cited first, 23 of 23)

- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Just Interpolate: Kernel "Ridgeless" Regression Can Generalize](https://mlsanity.com/paper/1808.00387/) (2018)
- [Multiplicative Normalizing Flows for Variational Bayesian Neural Networks](https://mlsanity.com/paper/1703.01961/) (2017)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [Classification vs regression in overparameterized regimes: Does the loss function matter?](https://mlsanity.com/paper/2005.08054/) (2020)
- [Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime](https://mlsanity.com/paper/2004.12019/) (2020)
- [Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfitting](https://mlsanity.com/paper/2106.09276/) (2021)
- [Dimension free ridge regression](https://mlsanity.com/paper/2210.08571/) (2022)
- [Scalable Bayes via Barycenter in Wasserstein Space](https://mlsanity.com/paper/1508.05880/) (2015)
- [Bayesian linear regression with sparse priors](https://mlsanity.com/paper/1403.0735/) (2014)
- [Using stacking to average Bayesian predictive distributions](https://mlsanity.com/paper/1704.02030/) (2017)
- [Towards an Understanding of Benign Overfitting in Neural Networks](https://mlsanity.com/paper/2106.03212/) (2021)
- [Foolish Crowds Support Benign Overfitting](https://mlsanity.com/paper/2110.02914/) (2021)
- [Tight bounds for minimum l1-norm interpolation of noisy data](https://mlsanity.com/paper/2111.05987/) (2021)
- [Empirical Bayes posterior concentration in sparse high-dimensional linear models](https://mlsanity.com/paper/1406.7718/) (2014)
- [Bayesian Bootstrap Spike-and-Slab LASSO](https://mlsanity.com/paper/2011.14279/) (2020)
- [Bayesian methods for low-rank matrix estimation: short survey and theoretical study](https://mlsanity.com/paper/1306.3862/) (2013)
- [Triple the gamma -- A unifying shrinkage prior for variance and variable selection in sparse state space and TVP models](https://mlsanity.com/paper/1912.03100/) (2019)
- [Learning Neural Networks with Adaptive Regularization](https://mlsanity.com/paper/1907.06288/) (2019)
- [Benign Overfitting in Time Series Linear Models with Over-Parameterization](https://mlsanity.com/paper/2204.08369/) (2022)
- [LR-GLM: High-Dimensional Bayesian Inference Using Low-Rank Data Approximations](https://mlsanity.com/paper/1905.07499/) (2019)
- [High-Dimensional Multivariate Posterior Consistency Under Global-Local Shrinkage Priors](https://mlsanity.com/paper/1711.07635/) (2017)

## Abstract (arXiv)

In high-dimensional Bayesian statistics, various methods have been developed, including prior distributions that induce parameter sparsity to handle many parameters. Yet, these approaches often overlook the rich spectral structure of the covariate matrix, which can be crucial when true signals are not sparse. To address this gap, we introduce a data-adaptive Gaussian prior whose covariance is aligned with the leading eigenvectors of the sample covariance. This prior design targets the data's intrinsic complexity rather than its ambient dimension by concentrating the parameter search along principal data directions. We establish contraction rates of the corresponding posterior distribution, which reveal how the mass in the spectrum affects the prediction error bounds. Furthermore, we derive a truncated Gaussian approximation to the posterior (i.e., a Bernstein-von Mises-type result), which allows for uncertainty quantification with a reduced computational burden. Our findings demonstrate that Bayesian methods leveraging spectral information of the data are effective for estimation in non-sparse, high-dimensional settings.