mlsanityData to 14 Oct 2026

Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It

arXiv 1412.3730 ↗First posted 11 Dec 2014math.ST stat.ME stat.THcited by 95

Not on the map. Why?

Abstract (arXiv)

We empirically show that Bayesian inference can be inconsistent under misspecification in simple linear regression problems, both in a model averaging/selection and in a Bayesian ridge regression setting. We use the standard linear model, which assumes homoskedasticity, whereas the data are heteroskedastic, and observe that the posterior puts its mass on ever more high-dimensional models as the sample size increases. To remedy the problem, we equip the likelihood in Bayes' theorem with an exponent called the learning rate, and we propose the Safe Bayesian method to learn the learning rate from the data. SafeBayes tends to select small learning rates as soon the standard posterior is not `cumulatively concentrated', and its results on our data are quite encouraging.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 14; 2023: 20; 2024: 14; 2025: 12; 2026: 21.

95 papers.

Sort:

  1. 2021User-friendly introduction to PAC-Bayes boundscited by 239
  2. 2023Modern Bayesian Experimental Designcited by 125
  3. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 104
  4. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  5. 2021The Shape of Learning Curves: a Reviewcited by 70
  6. 2024Variational Bayesian Last Layerscited by 69
  7. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  8. 2021Robust Generalised Bayesian Inference for Intractable Likelihoodscited by 53
  9. 2021Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networkscited by 49
  10. 2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
  11. 2022Robust Neural Posterior Estimation and Statistical Model Criticismcited by 46
  12. 2022Robust Bayesian Inference for Simulator-based Models via the MMD Posterior Bootstrapcited by 41
38 more
  1. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  2. 2022Learning Causal Representations of Single Cells via Sparse Mechanism Shift Modelingcited by 34
  3. 2023FIND: A Function Description Benchmark for Evaluating Interpretability Methodscited by 32
  4. 2022Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimationcited by 28
  5. 2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
  6. 2024Outlier-robust Kalman Filtering through Generalised Bayescited by 21
  7. 2023Adversarial robustness of amortized Bayesian inferencecited by 17
  8. 2023Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimationcited by 17
  9. 2022The no-free-lunch theorems of supervised learningcited by 16
  10. 2021On the Robustness to Misspecification of α-Posteriors and Their Variational Approximationscited by 15
  11. 2022How Tempering Fixes Data Augmentation in Bayesian Neural Networkscited by 13
  12. 2023A Novel Bayes' Theorem for Upper Probabilitiescited by 13
  13. 2025Simulation-based Bayesian inference under model misspecificationcited by 9
  14. 2023A Unifying Perspective on Non-Stationary Kernels for Deeper Gaussian Processescited by 8
  15. 2024Bayesian Neural Networks with Domain Knowledge Priorscited by 8
  16. 2024Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks: An Extended Investigationcited by 8
  17. 2024Concentration of a sparse Bayesian model with Horseshoe prior in estimating high-dimensional precision matrixcited by 8
  18. 2021Comparing the Value of Labeled and Unlabeled Data in Method-of-Moments Latent Variable Estimationcited by 7
  19. 2023Metrics for Bayesian Optimal Experiment Design under Model Misspecificationcited by 7
  20. 2023On the Properties and Estimation of Pointwise Mutual Information Profilescited by 7
  21. 2024Concentration properties of fractional posterior in 1-bit matrix completioncited by 7
  22. 2022Robust PACm: Training Ensemble Models Under Misspecification and Outlierscited by 6
  23. 2024A sparse PAC-Bayesian approach for high-dimensional quantile predictioncited by 6
  24. 2021Active multi-fidelity Bayesian online changepoint detectioncited by 5
  25. 2022Variational Inference of overparameterized Bayesian Neural Networks: a theoretical and empirical studycited by 5
  26. 2022Meta-Uncertainty in Bayesian Model Comparisoncited by 5
  27. 2025Active Learning of Model Discrepancy with Bayesian Experimental Designcited by 5
  28. 2022Tuning Stochastic Gradient Algorithms for Statistical Inference via Large-Sample Asymptoticscited by 4
  29. 2023Semiparametric inference using fractional posteriorscited by 4
  30. 2023On the Temperature of Bayesian Graph Neural Networks for Conformal Predictioncited by 4
  31. 2022Cold Posteriors through PAC-Bayescited by 3
  32. 2023Multi-Predictor Fusion: Combining Learning-based and Rule-based Trajectory Predictorscited by 3
  33. 2023Information-theoretic Analysis of Test Data Sensitivity in Uncertaintycited by 3
  34. 2023The fine print on tempered posteriorscited by 3
  35. 2024Predictive variational inference: Learn the predictively optimal posterior distributioncited by 3
  36. 2021Posterior concentration and fast convergence rates for generalized Bayesian learningcited by 2
  37. 2021Uncertainty estimation under model misspecification in neural network regressioncited by 2
  38. 2022Scalable Semi-Modular Inference with Variational Meta-Posteriorscited by 2

50 most cited of 95 All 95 (JSON)