mlsanityData to 7 Oct 2026

Bayesian Model Selection, the Marginal Likelihood, and Generalization

arXiv 2202.11678 ↗First posted 23 Feb 2022cs.LG stat.MLcited by 56

Not on the map. Why?

Abstract (arXiv)

How do we compare between hypotheses that are entirely consistent with observations? The marginal likelihood (aka Bayesian evidence), which represents the probability of generating our observations from a prior, provides a distinctive approach to this foundational question, automatically encoding Occam's razor. Although it has been observed that the marginal likelihood can overfit and is sensitive to prior assumptions, its limitations for hyperparameter learning and discrete model comparison have not been thoroughly investigated. We first revisit the appealing properties of the marginal likelihood for learning constraints and hypothesis testing. We then highlight the conceptual and practical issues in using the marginal likelihood as a proxy for generalization. Namely, we show how marginal likelihood can be negatively correlated with generalization, with implications for neural architecture search, and can lead to both underfitting and overfitting in hyperparameter learning. We also re-examine the connection between the marginal likelihood and PAC-Bayes bounds and use this connection to further elucidate the shortcomings of the marginal likelihood for model selection. We provide a partial remedy through a conditional marginal likelihood, which we show is more aligned with generalization, and practically valuable for large-scale hyperparameter learning, such as in deep kernel learning.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 45 of its 101 references that are in the map. First posted on arXiv 23 Feb 2022.

Builds on

45 of 101 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2015Variational Inference with Normalizing Flowscited by 2,161
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  10. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  11. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  12. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
33 more
  1. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  2. 2015Deep Kernel Learningcited by 440
  3. 2013Gaussian Processes for Big Datacited by 377
  4. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  5. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  6. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  7. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  8. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  9. 2013Gaussian Process Kernels for Pattern Discovery and Extrapolationcited by 131
  10. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  11. 2021Bayesian Neural Network Priors Revisitedcited by 121
  12. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  13. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  14. 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
  15. 2020On the Benefits of Invariance in Neural Networkscited by 78
  16. 2020On the role of data in PAC-Bayes boundscited by 64
  17. 2016A Strongly Quasiconvex PAC-Bayesian Boundcited by 54
  18. 2013Understanding predictive information criteria for Bayesian modelscited by 51
  19. 2018Learning Invariances using the Marginal Likelihoodcited by 51
  20. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 50
  21. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  22. 2014Automatic Construction and Natural-Language Description of Nonparametric Regression Modelscited by 40
  23. 2019On the marginal likelihood and cross-validationcited by 36
  24. 2021Differentiable Annealed Importance Sampling and the Perils of Gradient Noisecited by 36
  25. 2020Marginal likelihood computation for model selection and hypothesis testing: an extensive reviewcited by 33
  26. 2021From Predictions to Decisions: The Importance of Joint Predictive Distributionscited by 28
  27. 2021The Neural Testbed: Evaluating Joint Predictionscited by 22
  28. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  29. 2015Sandwiching the marginal likelihood using bidirectional Monte Carlocited by 20
  30. 2020A Bayesian Perspective on Training Speed and Model Selectioncited by 19
  31. 2020PACm-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regimecited by 15
  32. 2017Improving Output Uncertainty Estimation and Generalization in Deep Learning via Neural Network Gaussian Processescited by 14
  33. 2015The Human Kernelcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 16; 2024: 10; 2025: 12; 2026: 6.

56 papers.

Sort:

By area:

  1. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  2. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  3. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  4. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  5. 2022GFlowOut: Dropout with Generative Flow Networkscited by 22
  6. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  7. 2022Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networkscited by 14
  8. 2023Leveraging Self-Consistency for Data-Efficient Amortized Bayesian Inferencecited by 14
  9. 2023A Deep Learning Method for Comparing Bayesian Hierarchical Modelscited by 11
  10. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 10
  11. 2023Approximately Bayes-Optimal Pseudo Label Selectioncited by 10
  12. 2024PAC-Bayes-Chernoff bounds for unbounded lossescited by 9
38 more
  1. 2022Are you using test log-likelihood correctly?cited by 8
  2. 2023Hyperparameter Optimization through Neural Network Partitioningcited by 8
  3. 2024Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks: An Extended Investigationcited by 8
  4. 2022Understanding Deep Learning via Decision Boundarycited by 7
  5. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
  6. 2025Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Trackcited by 7
  7. 2023Guided Deep Kernel Learningcited by 6
  8. 2024Bayesian Exploration of Pre-trained Models for Low-shot Image Classificationcited by 6
  9. 2022Gaussian Process Surrogate Models for Neural Networkscited by 5
  10. 2022Meta-Uncertainty in Bayesian Model Comparisoncited by 5
  11. 2023Sharp Calibrated Gaussian Processescited by 5
  12. 2023Approximate Bayesian Class-Conditional Models under Continuous Representation Shiftcited by 5
  13. 2025Improving the Accuracy of Amortized Model Comparison with Self-Consistencycited by 5
  14. 2023TIC-TAC: A Framework for Improved Covariance Estimation in Deep Heteroscedastic Regressioncited by 4
  15. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  16. 2022Marginal and Joint Cross-Entropies & Predictives for Online Bayesian Inference, Active Learning, and Active Samplingcited by 3
  17. 2022Cold Posteriors through PAC-Bayescited by 3
  18. 2023Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Modelscited by 3
  19. 2024Preventing Model Collapse in Gaussian Process Latent Variable Modelscited by 3
  20. 2025Automated Model Discovery via Multi-modal & Multi-step Pipelinecited by 3
  21. 2025Uncertainty-Aware Diagnostics for Physics-Informed Machine Learningcited by 3
  22. 2026Position: agentic AI orchestration should be Bayes-consistentcited by 3
  23. 2022R'enyi Divergence Deep Mutual Learningcited by 2
  24. 2024Data-Agnostic Face Image Synthesis Detection Using Bayesian CNNscited by 2
  25. 2025Learning Hyperparameters via a Data-Emphasized Variational Objectivecited by 2
  26. 2025Learning Decision Trees as Amortized Structure Inferencecited by 2
  27. 2022An Empirical Analysis of the Advantages of Finite- v.s. Infinite-Width Bayesian Neural Networkscited by 1
  28. 2023Bootstrap Your Own Variancecited by 1
  29. 2024Learning the Regularization Strength for Deep Fine-Tuning via a Data-Emphasized Variational Objectivecited by 1
  30. 2025On Equivariant Model Selection through the Lens of Uncertaintycited by 1
  31. 2025LVM-GP: Uncertainty-Aware PDE Solver via coupling latent variable model and Gaussian processcited by 1
  32. 2026Scalable Simulation-Based Model Inference with Test-Time Complexity Controlcited by 1
  33. 2023In all LikelihoodS: How to Reliably Select Pseudo-Labeled Data for Self-Training in Semi-Supervised Learningcited by 0
  34. 2023Gibbs-Based Information Criteria and the Over-Parameterized Regimecited by 0
  35. 2023Amortised Inference in Bayesian Neural Networkscited by 0
  36. 2024Detecting Face Synthesis Using a Concealed Fusion Modelcited by 0
  37. 2024Gaussian-process-regression-based method for the localization of exceptional points in complex resonance spectracited by 0
  38. 2025Identifying Information from Observations with Uncertainty and Noveltycited by 0

50 most cited of 56 All 56 (JSON)