# On the Importance of Strong Baselines in Bayesian Deep Learning

arXiv: https://arxiv.org/abs/1811.09385 · HTML: https://mlsanity.com/paper/1811.09385/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2018.

## Built on by (most cited first, 12 of 12)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Flexible Model Aggregation for Quantile Regression](https://mlsanity.com/paper/2103.00083/) (2021)
- [NOMU: Neural Optimization-based Model Uncertainty](https://mlsanity.com/paper/2102.13640/) (2021)
- [A Holistic Assessment of the Reliability of Machine Learning Systems](https://mlsanity.com/paper/2307.10586/) (2023)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [Personalized Benchmarking with the Ludwig Benchmarking Toolkit](https://mlsanity.com/paper/2111.04260/) (2021)
- [Using AntiPatterns to avoid MLOps Mistakes](https://mlsanity.com/paper/2107.00079/) (2021)
- [Bayesian Active Summarization](https://mlsanity.com/paper/2110.04480/) (2021)
- [Revisiting the Effects of Stochasticity for Hamiltonian Samplers](https://mlsanity.com/paper/2106.16200/) (2021)
- [ELBOing Stein: Variational Bayes with Stein Mixture Inference](https://mlsanity.com/paper/2410.22948/) (2024)
- [Previously on... Automating Code Review](https://mlsanity.com/paper/2508.18003/) (2025)
- [FineMorphs: Affine-diffeomorphic sequences for regression](https://mlsanity.com/paper/2305.17255/) (2023)

## Abstract (arXiv)

Like all sub-fields of machine learning Bayesian Deep Learning is driven by empirical validation of its theoretical proposals. Given the many aspects of an experiment it is always possible that minor or even major experimental flaws can slip by both authors and reviewers. One of the most popular experiments used to evaluate approximate inference techniques is the regression experiment on UCI datasets. However, in this experiment, models which have been trained to convergence have often been compared with baselines trained only for a fixed number of iterations. We find that a well-established baseline, Monte Carlo dropout, when evaluated under the same experimental settings shows significant improvements. In fact, the baseline outperforms or performs competitively with methods that claimed to be superior to the very same baseline method when they were introduced. Hence, by exposing this flaw in experimental procedure, we highlight the importance of using identical experimental setups to evaluate, compare, and benchmark methods in Bayesian Deep Learning.