Data to 14 Oct 2026

Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning

arXiv 2104.04975 ↗First posted 11 Apr 2021stat.ML cs.LGcited by 106

Not on the map. Why?

Abstract (arXiv)

Marginal-likelihood based model-selection, even though promising, is rarely used in deep learning due to estimation difficulties. Instead, most approaches rely on validation data, which may not be readily available. In this work, we present a scalable marginal-likelihood estimation method to select both hyperparameters and network architectures, based on the training data alone. Some hyperparameters can be estimated online during training, simplifying the procedure. Our marginal-likelihood estimate is based on Laplace's method and Gauss-Newton approximations to the Hessian, and it outperforms cross-validation and manual-tuning on standard regression and image classification datasets, especially in terms of calibration and out-of-distribution detection. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable (e.g., in nonstationary settings).

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Builds on 30 of its 53 references that are in the map. First posted on arXiv 11 Apr 2021.

Builds on

30 of 53 references.

References · 30

2015Deep Residual Learning for Image Recognitioncited by 52,096
2014Adam: A Method for Stochastic Optimizationcited by 37,240
2017On Calibration of Modern Neural Networkscited by 4,960
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
2016Wide Residual Networkscited by 3,070
2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
2015Weight Uncertainty in Neural Networkscited by 554
2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
2019Fantastic Generalization Measures and Where to Find Themcited by 513
2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295

18 more

2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
2019Practical Deep Learning with Bayesian Principlescited by 158
2017Practical Gauss-Newton Optimisation for Deep Learningcited by 142
2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
2018Three Mechanisms of Weight Decay Regularizationcited by 113
2017Noisy Natural Gradient as Variational Inferencecited by 94
2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 80
2019BackPACK: Packing more into backpropcited by 78
2018Learning Invariances using the Marginal Likelihoodcited by 51
2019DeepOBS: A Deep Learning Optimizer Benchmark Suitecited by 37
2019On the marginal likelihood and cross-validationcited by 37
2020Marginal likelihood computation for model selection and hypothesis testing: an extensive reviewcited by 33
2019Bayesian Image Classification with Deep Convolutional Gaussian Processescited by 21
2020A Bayesian Perspective on Training Speed and Model Selectioncited by 19
2013The divergence of the BFGS and Gauss Newton Methodscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 19; 2022: 17; 2023: 24; 2024: 19; 2025: 16; 2026: 11.

106 papers.

Sort:

By area:

2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
2021Repulsive Deep Ensembles are Bayesiancited by 104
2021Priors in Bayesian Deep Learning: A Reviewcited by 101
2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 57
2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
2024Rethinking Aleatoric and Epistemic Uncertaintycited by 42
2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29

38 more

2021Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubscited by 28
2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
2022Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learningcited by 26
2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
2022Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEscited by 25
2023Learning Layer-wise Equivariances Automatically using Gradientscited by 23
2025Generative Uncertainty in Diffusion Modelscited by 23
2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
2021On Stein Variational Neural Network Ensemblescited by 21
2021BNNpriors: A library for Bayesian neural network inference with different prior distributionscited by 20
2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
2021Neural Variational Gradient Descentcited by 19
2023Fixing Overconfidence in Dynamic Neural Networkscited by 19
2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
2024Reparameterization invariance in approximate Bayesian inferencecited by 17
2022Learning Invariant Weights in Neural Networkscited by 16
2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16
2023Improving Neural Additive Models with Bayesian Principlescited by 15
2023Riemannian Laplace approximations for Bayesian neural networkscited by 15
2022Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networkscited by 14
2022Deep Active Learning with Noise Stabilitycited by 14
2021Pathologies in priors and inference for Bayesian transformerscited by 13
2024Post-hoc Probabilistic Vision-Language Modelscited by 13
2021Probing as Quantifying Inductive Biascited by 12
2024Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learningcited by 11
2024FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learningcited by 11
2021Model Selection for Bayesian Autoencoderscited by 10
2022Uncertainty Estimation for Computed Tomography with a Linearised Deep Image Priorcited by 10
2022Laplacian Autoencoders for Learning Stochastic Representationscited by 10
2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
2021Deep Classifiers with Label Noise Modeling and Distance Awarenesscited by 8
2023Hyperparameter Optimization through Neural Network Partitioningcited by 8
2024Function-space Parameterization of Neural Networks for Sequential Learningcited by 8

50 most cited of 106 All 106 (JSON)