# Benchmarking the Neural Linear Model for Regression

arXiv: https://arxiv.org/abs/1912.08416 · HTML: https://mlsanity.com/paper/1912.08416/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2019.

## Built on by (most cited first, 30 of 31)

- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [Do Bayesian Neural Networks Need To Be Fully Stochastic?](https://mlsanity.com/paper/2211.06291/) (2022)
- [A Framework and Benchmark for Deep Batch Active Learning for Regression](https://mlsanity.com/paper/2203.09410/) (2022)
- [Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning](https://mlsanity.com/paper/2111.03577/) (2021)
- [Bayesian Deep Basis Fitting for Depth Completion with Uncertainty](https://mlsanity.com/paper/2103.15254/) (2021)
- [NOMU: Neural Optimization-based Model Uncertainty](https://mlsanity.com/paper/2102.13640/) (2021)
- [Bayesian Optimization via Continual Variational Last Layer Training](https://mlsanity.com/paper/2412.09477/) (2024)
- [Trust Your Robots! Predictive Uncertainty Estimation of Neural Networks with Sparse Gaussian Processes](https://mlsanity.com/paper/2109.09690/) (2021)
- [Improved uncertainty quantification for neural networks with Bayesian last layer](https://mlsanity.com/paper/2302.10975/) (2023)
- [A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2310.08287/) (2023)
- [Bayesian Optimization-based Combinatorial Assignment](https://mlsanity.com/paper/2208.14698/) (2022)
- [Incorporating Unlabelled Data into Bayesian Neural Networks](https://mlsanity.com/paper/2304.01762/) (2023)
- [Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics](https://mlsanity.com/paper/2303.05101/) (2023)
- [Towards a Unified Framework for Uncertainty-aware Nonlinear Variable Selection with Theoretical Guarantees](https://mlsanity.com/paper/2204.07293/) (2022)
- [Density Uncertainty Layers for Reliable Uncertainty Estimation](https://mlsanity.com/paper/2306.12497/) (2023)
- [On Feynman--Kac training of partial Bayesian neural networks](https://mlsanity.com/paper/2310.19608/) (2023)
- [Spectral methods: crucial for machine learning, natural for quantum computers?](https://mlsanity.com/paper/2603.24654/) (2026)
- [Uncertainty Quantification in Neural Differential Equations](https://mlsanity.com/paper/2111.04207/) (2021)
- [Marginally calibrated response distributions for end-to-end learning in autonomous driving](https://mlsanity.com/paper/2110.01050/) (2021)
- [Achieving Well-Informed Decision-Making in Drug Discovery: A Comprehensive Calibration Study using Neural Network-Based Structure-Activity Models](https://mlsanity.com/paper/2407.14185/) (2024)
- [Enhancing Uncertainty Estimation and Interpretability via Bayesian Non-negative Decision Layer](https://mlsanity.com/paper/2505.22199/) (2025)
- [Probabilistic Unrolling: Scalable, Inverse-Free Maximum Likelihood Estimation for Latent Gaussian Models](https://mlsanity.com/paper/2306.03249/) (2023)
- [Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence](https://mlsanity.com/paper/2507.20272/) (2025)
- [Understanding Uncertainty in Bayesian Deep Learning](https://mlsanity.com/paper/2106.13055/) (2021)
- [FineMorphs: Affine-diffeomorphic sequences for regression](https://mlsanity.com/paper/2305.17255/) (2023)
- [Multivariate Bayesian Last Layer for Regression with Uncertainty Quantification and Decomposition](https://mlsanity.com/paper/2405.01761/) (2024)
- [Structured Partial Stochasticity in Bayesian Neural Networks](https://mlsanity.com/paper/2405.17666/) (2024)
- [Stein Variational Newton Neural Network Ensembles](https://mlsanity.com/paper/2411.01887/) (2024)
- [Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding](https://mlsanity.com/paper/2605.08446/) (2026)

## Abstract (arXiv)

The neural linear model is a simple adaptive Bayesian linear regression method that has recently been used in a number of problems ranging from Bayesian optimization to reinforcement learning. Despite its apparent successes in these settings, to the best of our knowledge there has been no systematic exploration of its capabilities on simple regression tasks. In this work we characterize these on the UCI datasets, a popular benchmark for Bayesian regression models, as well as on the recently introduced UCI "gap" datasets, which are better tests of out-of-distribution uncertainty. We demonstrate that the neural linear model is a simple method that shows generally good performance on these tasks, but at the cost of requiring good hyperparameter tuning.