# On the marginal likelihood and cross-validation

arXiv: https://arxiv.org/abs/1905.08737 · HTML: https://mlsanity.com/paper/1905.08737/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2019.

## Built on by (most cited first, 30 of 37)

- [Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning](https://mlsanity.com/paper/2104.04975/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Bayesian Model Selection, the Marginal Likelihood, and Generalization](https://mlsanity.com/paper/2202.11678/) (2022)
- [In-Context Learning Learns Label Relationships but Is Not Conventional Learning](https://mlsanity.com/paper/2307.12375/) (2023)
- [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](https://mlsanity.com/paper/2202.10638/) (2022)
- [Rethinking Aleatoric and Epistemic Uncertainty](https://mlsanity.com/paper/2412.20892/) (2024)
- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [The Neural Process Family: Survey, Applications and Perspectives](https://mlsanity.com/paper/2209.00517/) (2022)
- [The no-free-lunch theorems of supervised learning](https://mlsanity.com/paper/2202.04513/) (2022)
- [Learning Invariant Weights in Neural Networks](https://mlsanity.com/paper/2202.12439/) (2022)
- [Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posed](https://mlsanity.com/paper/2203.09179/) (2022)
- [Probing as Quantifying Inductive Bias](https://mlsanity.com/paper/2110.08388/) (2021)
- [The Interpolating Information Criterion for Overparameterized Models](https://mlsanity.com/paper/2307.07785/) (2023)
- [Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes](https://mlsanity.com/paper/2210.07612/) (2022)
- [Hyperparameter Optimization through Neural Network Partitioning](https://mlsanity.com/paper/2304.14766/) (2023)
- [Scalable Cross Validation Losses for Gaussian Process Models](https://mlsanity.com/paper/2105.11535/) (2021)
- [Revisiting Active Sets for Gaussian Process Decoders](https://mlsanity.com/paper/2209.04636/) (2022)
- [Meta-Uncertainty in Bayesian Model Comparison](https://mlsanity.com/paper/2210.07278/) (2022)
- [Sharp Calibrated Gaussian Processes](https://mlsanity.com/paper/2302.11961/) (2023)
- [On Masked Pre-training and the Marginal Likelihood](https://mlsanity.com/paper/2306.00520/) (2023)
- [Enhancing Transfer Learning with Flexible Nonparametric Posterior Sampling](https://mlsanity.com/paper/2403.07282/) (2024)
- [Efficient computation of the volume of a polytope in high-dimensions using Piecewise Deterministic Markov Processes](https://mlsanity.com/paper/2202.09129/) (2022)
- [Bayesian Regression Markets](https://mlsanity.com/paper/2310.14992/) (2023)
- [Distributional Training Data Attribution: What do Influence Functions Sample?](https://mlsanity.com/paper/2506.12965/) (2025)
- [Asymptotic Bounds for Smoothness Parameter Estimates in Gaussian Process Interpolation](https://mlsanity.com/paper/2203.05400/) (2022)
- [clusterBMA: Bayesian model averaging for clustering](https://mlsanity.com/paper/2209.04117/) (2022)
- [Spectral information criterion for automatic elbow detection](https://mlsanity.com/paper/2308.09108/) (2023)
- [Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference](https://mlsanity.com/paper/2402.03175/) (2024)
- [Fine-Tuned In-Context Learners for Efficient Adaptation](https://mlsanity.com/paper/2512.19879/) (2025)

## Abstract (arXiv)

In Bayesian statistics, the marginal likelihood, also known as the evidence, is used to evaluate model fit as it quantifies the joint probability of the data under the prior. In contrast, non-Bayesian models are typically compared using cross-validation on held-out data, either through k-fold partitioning or leave-p-out subsampling. We show that the marginal likelihood is formally equivalent to exhaustive leave-p-out cross-validation averaged over all values of p and all held-out test sets when using the log posterior predictive probability as the scoring rule. Moreover, the log posterior predictive is the only coherent scoring rule under data exchangeability. This offers new insight into the marginal likelihood and cross-validation and highlights the potential sensitivity of the marginal likelihood to the choice of the prior. We suggest an alternative approach using cumulative cross-validation following a preparatory training phase. Our work has connections to prequential analysis and intrinsic Bayes factors but is motivated through a different course.