# Performance of Bayesian linear regression in a model with mismatch

arXiv: https://arxiv.org/abs/2107.06936 · HTML: https://mlsanity.com/paper/2107.06936/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 58 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 15 of 15)

- [Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models](https://mlsanity.com/paper/1708.03395/) (2017)
- [A modern maximum-likelihood theory for high-dimensional logistic regression](https://mlsanity.com/paper/1803.06964/) (2018)
- [Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization](https://mlsanity.com/paper/2006.06560/) (2020)
- [Variational Bayes for high-dimensional linear regression with sparse priors](https://mlsanity.com/paper/1904.07150/) (2019)
- [Bayesian linear regression with sparse priors](https://mlsanity.com/paper/1403.0735/) (2014)
- [Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)](https://mlsanity.com/paper/2006.06581/) (2020)
- [All-or-nothing statistical and computational phase transitions in sparse spiked matrix estimation](https://mlsanity.com/paper/2006.07971/) (2020)
- [Statistical Mechanics of High-Dimensional Inference](https://mlsanity.com/paper/1601.04650/) (2016)
- [Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensions](https://mlsanity.com/paper/2006.08917/) (2020)
- [High-Dimensional Regression with Binary Coefficients. Estimating Squared Error and a Phase Transition](https://mlsanity.com/paper/1701.04455/) (2017)
- [Empirical Bayes posterior concentration in sparse high-dimensional linear models](https://mlsanity.com/paper/1406.7718/) (2014)
- [Variational Inference in high-dimensional linear regression](https://mlsanity.com/paper/2104.12232/) (2021)
- [Asymptotic errors for convex penalized linear regression beyond Gaussian matrices](https://mlsanity.com/paper/2002.04372/) (2020)
- [Information theoretic limits of learning a sparse rule](https://mlsanity.com/paper/2006.11313/) (2020)
- [An equivalence between high dimensional Bayes optimal inference and M-estimation](https://mlsanity.com/paper/1609.07060/) (2016)

## Built on by (most cited first, 6 of 6)

- [Contrasting random and learned features in deep Bayesian linear regression](https://mlsanity.com/paper/2203.00573/) (2022)
- [On double-descent in uncertainty quantification in overparametrized models](https://mlsanity.com/paper/2210.12760/) (2022)
- [The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?](https://mlsanity.com/paper/2205.10009/) (2022)
- [Near-optimal multiple testing in Bayesian linear models with finite-sample FDR control](https://mlsanity.com/paper/2211.02778/) (2022)
- [Unrolled denoising networks provably learn optimal Bayesian inference](https://mlsanity.com/paper/2409.12947/) (2024)
- [Bayesian Inference with Nonlinear Generative Models: Comments on Secure Learning](https://mlsanity.com/paper/2201.09986/) (2022)

## Abstract (arXiv)

In this paper we analyze, for a model of linear regression with gaussian covariates, the performance of a Bayesian estimator given by the mean of a log-concave posterior distribution with gaussian prior, in the high-dimensional limit where the number of samples and the covariates' dimension are large and proportional. Although the high-dimensional analysis of Bayesian estimators has been previously studied for Bayesian-optimal linear regression where the correct posterior is used for inference, much less is known when there is a mismatch. Here we consider a model in which the responses are corrupted by gaussian noise and are known to be generated as linear combinations of the covariates, but the distributions of the ground-truth regression coefficients and of the noise are unknown. This regression task can be rephrased as a statistical mechanics model known as the Gardner spin glass, an analogy which we exploit. Using a leave-one-out approach we characterize the mean-square error for the regression coefficients. We also derive the log-normalizing constant of the posterior. Similar models have been studied by Shcherbina and Tirozzi and by Talagrand, but our arguments are much more straightforward. An interesting consequence of our analysis is that in the quadratic loss case, the performance of the Bayesian estimator is independent of a global "temperature" hyperparameter and matches the ridge estimator: sampling and optimizing are equally good.