# Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It

arXiv: https://arxiv.org/abs/1412.3730 · HTML: https://mlsanity.com/paper/1412.3730/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2014.

## Built on by (most cited first, 30 of 95)

- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [Modern Bayesian Experimental Design](https://mlsanity.com/paper/2302.14545/) (2023)
- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Robust Generalised Bayesian Inference for Intractable Likelihoods](https://mlsanity.com/paper/2104.07359/) (2021)
- [Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks](https://mlsanity.com/paper/2112.08866/) (2021)
- [On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification](https://mlsanity.com/paper/2203.16481/) (2022)
- [Robust Neural Posterior Estimation and Statistical Model Criticism](https://mlsanity.com/paper/2210.06564/) (2022)
- [Robust Bayesian Inference for Simulator-based Models via the MMD Posterior Bootstrap](https://mlsanity.com/paper/2202.04744/) (2022)
- [Data augmentation in Bayesian neural networks and the cold posterior effect](https://mlsanity.com/paper/2106.05586/) (2021)
- [Learning Causal Representations of Single Cells via Sparse Mechanism Shift Modeling](https://mlsanity.com/paper/2211.03553/) (2022)
- [FIND: A Function Description Benchmark for Evaluating Interpretability Methods](https://mlsanity.com/paper/2309.03886/) (2023)
- [Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation](https://mlsanity.com/paper/2208.13624/) (2022)
- [Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect](https://mlsanity.com/paper/2106.06596/) (2021)
- [Outlier-robust Kalman Filtering through Generalised Bayes](https://mlsanity.com/paper/2405.05646/) (2024)
- [Adversarial robustness of amortized Bayesian inference](https://mlsanity.com/paper/2305.14984/) (2023)
- [Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimation](https://mlsanity.com/paper/2305.15208/) (2023)
- [The no-free-lunch theorems of supervised learning](https://mlsanity.com/paper/2202.04513/) (2022)
- [On the Robustness to Misspecification of α-Posteriors and Their Variational Approximations](https://mlsanity.com/paper/2104.08324/) (2021)
- [How Tempering Fixes Data Augmentation in Bayesian Neural Networks](https://mlsanity.com/paper/2205.13900/) (2022)
- [A Novel Bayes' Theorem for Upper Probabilities](https://mlsanity.com/paper/2307.06831/) (2023)
- [Simulation-based Bayesian inference under model misspecification](https://mlsanity.com/paper/2503.12315/) (2025)
- [A Unifying Perspective on Non-Stationary Kernels for Deeper Gaussian Processes](https://mlsanity.com/paper/2309.10068/) (2023)
- [Bayesian Neural Networks with Domain Knowledge Priors](https://mlsanity.com/paper/2402.13410/) (2024)
- [Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks: An Extended Investigation](https://mlsanity.com/paper/2406.03154/) (2024)
- [Concentration of a sparse Bayesian model with Horseshoe prior in estimating high-dimensional precision matrix](https://mlsanity.com/paper/2406.14269/) (2024)
- [Comparing the Value of Labeled and Unlabeled Data in Method-of-Moments Latent Variable Estimation](https://mlsanity.com/paper/2103.02761/) (2021)

## Abstract (arXiv)

We empirically show that Bayesian inference can be inconsistent under misspecification in simple linear regression problems, both in a model averaging/selection and in a Bayesian ridge regression setting. We use the standard linear model, which assumes homoskedasticity, whereas the data are heteroskedastic, and observe that the posterior puts its mass on ever more high-dimensional models as the sample size increases. To remedy the problem, we equip the likelihood in Bayes' theorem with an exponent called the learning rate, and we propose the Safe Bayesian method to learn the learning rate from the data. SafeBayes tends to select small learning rates as soon the standard posterior is not `cumulatively concentrated', and its results on our data are quite encouraging.