# On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks

arXiv: https://arxiv.org/abs/2203.09168 · HTML: https://mlsanity.com/paper/2203.09168/

## Key facts

- Cited by 120 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Causal Influence Detection for Improving Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2106.03443/) (2021)
- [Reliable training and estimation of variance networks](https://mlsanity.com/paper/1906.03260/) (2019)
- [Structure-Aware Residual Pyramid Network for Monocular Depth Estimation](https://mlsanity.com/paper/1907.06023/) (2019)
- [Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors](https://mlsanity.com/paper/2101.05036/) (2021)
- [Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization](https://mlsanity.com/paper/2006.04910/) (2020)

## Built on by (most cited first, 30 of 120)

- [A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement](https://mlsanity.com/paper/2204.09308/) (2022)
- [Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2407.07532/) (2024)
- [Faithful Heteroscedastic Regression with Neural Networks](https://mlsanity.com/paper/2212.09184/) (2022)
- [Single-model uncertainty quantification in neural network potentials does not consistently outperform model ensembles](https://mlsanity.com/paper/2305.01754/) (2023)
- [Embrace the Gap: VAEs Perform Independent Mechanism Analysis](https://mlsanity.com/paper/2206.02416/) (2022)
- [UnCRtainTS: Uncertainty Quantification for Cloud Removal in Optical Satellite Time Series](https://mlsanity.com/paper/2304.05464/) (2023)
- [Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief](https://mlsanity.com/paper/2210.06692/) (2022)
- [Coherent Soft Imitation Learning](https://mlsanity.com/paper/2305.16498/) (2023)
- [Optimal Training of Mean Variance Estimation Neural Networks](https://mlsanity.com/paper/2302.08875/) (2023)
- [Autoinverse: Uncertainty Aware Inversion of Neural Networks](https://mlsanity.com/paper/2208.13780/) (2022)
- [Ambiguous Images With Human Judgments for Robust Visual Event Classification](https://mlsanity.com/paper/2210.03102/) (2022)
- [Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization](https://mlsanity.com/paper/2310.06984/) (2023)
- [Fully Bayesian VIB-DeepSSM](https://mlsanity.com/paper/2305.05797/) (2023)
- [Understanding Pathologies of Deep Heteroskedastic Regression](https://mlsanity.com/paper/2306.16717/) (2023)
- [A Benchmark on Uncertainty Quantification for Deep Learning Prognostics](https://mlsanity.com/paper/2302.04730/) (2023)
- [Position: Epistemic uncertainty estimation methods are fundamentally incomplete](https://mlsanity.com/paper/2505.23506/) (2025)
- [Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data](https://mlsanity.com/paper/2411.03561/) (2024)
- [C-DiffSET: Leveraging Latent Diffusion for SAR-to-EO Image Translation with Confidence-Guided Reliable Object Generation](https://mlsanity.com/paper/2411.10788/) (2024)
- [PRISM: PRior-guided Imagination Sampling in world Models](https://mlsanity.com/paper/2606.07974/) (2026)
- [Uncertainty Quantification in Machine Learning for Biosignal Applications -- A Review](https://mlsanity.com/paper/2312.09454/) (2023)
- [MotionMap: Representing Multimodality in Human Pose Forecasting](https://mlsanity.com/paper/2412.18883/) (2024)
- [Robust Estimation of Causal Heteroscedastic Noise Models](https://mlsanity.com/paper/2312.10102/) (2023)
- [Winner-takes-all learners are geometry-aware conditional density estimators](https://mlsanity.com/paper/2406.04706/) (2024)
- [GUPNet++: Geometry Uncertainty Propagation Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2310.15624/) (2023)
- [Action Inference by Maximising Evidence: Zero-Shot Imitation from Observation with World Models](https://mlsanity.com/paper/2312.02019/) (2023)
- [U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpening](https://mlsanity.com/paper/2412.06243/) (2024)
- [Denoising, Fast and Slow: Difficulty-Aware Adaptive Sampling for Image Generation](https://mlsanity.com/paper/2604.19141/) (2026)
- [Estimating Regression Predictive Distributions with Sample Networks](https://mlsanity.com/paper/2211.13724/) (2022)
- [TIC-TAC: A Framework for Improved Covariance Estimation in Deep Heteroscedastic Regression](https://mlsanity.com/paper/2310.18953/) (2023)
- [Identifying Drivers of Predictive Aleatoric Uncertainty](https://mlsanity.com/paper/2312.07252/) (2023)

## Abstract (arXiv)

Capturing aleatoric uncertainty is a critical part of many machine learning systems. In deep learning, a common approach to this end is to train a neural network to estimate the parameters of a heteroscedastic Gaussian distribution by maximizing the logarithm of the likelihood function under the observed data. In this work, we examine this approach and identify potential hazards associated with the use of log-likelihood in conjunction with gradient-based optimizers. First, we present a synthetic example illustrating how this approach can lead to very poor but stable parameter estimates. Second, we identify the culprit to be the log-likelihood loss, along with certain conditions that exacerbate the issue. Third, we present an alternative formulation, termed β-NLL, in which each data point's contribution to the loss is weighted by the β-exponentiated variance estimate. We show that using an appropriate β largely mitigates the issue in our illustrative example. Fourth, we evaluate this approach on a range of domains and tasks and show that it achieves considerable improvements and performs more robustly concerning hyperparameters, both in predictive RMSE and log-likelihood criteria.