# Reliable training and estimation of variance networks

arXiv: https://arxiv.org/abs/1906.03260 · HTML: https://mlsanity.com/paper/1906.03260/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2019.

## Built on by (most cited first, 30 of 63)

- [On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks](https://mlsanity.com/paper/2203.09168/) (2022)
- [Do Bayesian Neural Networks Need To Be Fully Stochastic?](https://mlsanity.com/paper/2211.06291/) (2022)
- [Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification](https://mlsanity.com/paper/2109.10254/) (2021)
- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors](https://mlsanity.com/paper/2101.05036/) (2021)
- [Faithful Heteroscedastic Regression with Neural Networks](https://mlsanity.com/paper/2212.09184/) (2022)
- [UnCRtainTS: Uncertainty Quantification for Cloud Removal in Optical Satellite Time Series](https://mlsanity.com/paper/2304.05464/) (2023)
- [Pulling back information geometry](https://mlsanity.com/paper/2106.05367/) (2021)
- [Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networks](https://mlsanity.com/paper/2107.06068/) (2021)
- [Optimal Training of Mean Variance Estimation Neural Networks](https://mlsanity.com/paper/2302.08875/) (2023)
- [I Find Your Lack of Uncertainty in Computer Vision Disturbing](https://mlsanity.com/paper/2104.08188/) (2021)
- [Uncertainty Quantification via Neural Posterior Principal Components](https://mlsanity.com/paper/2309.15533/) (2023)
- [Bayesian Metric Learning for Uncertainty Quantification in Image Retrieval](https://mlsanity.com/paper/2302.01332/) (2023)
- [Understanding Pathologies of Deep Heteroskedastic Regression](https://mlsanity.com/paper/2306.16717/) (2023)
- [A Benchmark on Uncertainty Quantification for Deep Learning Prognostics](https://mlsanity.com/paper/2302.04730/) (2023)
- [Distributional Learning of Variational AutoEncoder: Application to Synthetic Data Generation](https://mlsanity.com/paper/2302.11294/) (2023)
- [Counterfactual Explanations via Riemannian Latent Space Traversal](https://mlsanity.com/paper/2411.02259/) (2024)
- [End-to-End Probabilistic Framework for Learning with Hard Constraints](https://mlsanity.com/paper/2506.07003/) (2025)
- [Energy-Based Generative Cooperative Saliency Prediction](https://mlsanity.com/paper/2106.13389/) (2021)
- [Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution](https://mlsanity.com/paper/2402.08383/) (2024)
- [Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise](https://mlsanity.com/paper/2406.03258/) (2024)
- [Robust Estimation of Causal Heteroscedastic Noise Models](https://mlsanity.com/paper/2312.10102/) (2023)
- [Decoder ensembling for learned latent geometries](https://mlsanity.com/paper/2408.07507/) (2024)
- [Probabilistic Machine Learning for Noisy Labels in Earth Observation](https://mlsanity.com/paper/2504.03478/) (2025)
- [Uncertainty Quantification for Regression using Proper Scoring Rules](https://mlsanity.com/paper/2509.26610/) (2025)
- [Deep Quantile Regression for Uncertainty Estimation in Unsupervised and Supervised Lesion Detection](https://mlsanity.com/paper/2109.09374/) (2021)
- [Estimating Regression Predictive Distributions with Sample Networks](https://mlsanity.com/paper/2211.13724/) (2022)
- [TIC-TAC: A Framework for Improved Covariance Estimation in Deep Heteroscedastic Regression](https://mlsanity.com/paper/2310.18953/) (2023)
- [Exploring Uncertainty in Deep Learning for Construction of Prediction Intervals](https://mlsanity.com/paper/2104.12953/) (2021)
- [Robust uncertainty estimates with out-of-distribution pseudo-inputs training](https://mlsanity.com/paper/2201.05890/) (2022)

## Abstract (arXiv)

We propose and investigate new complementary methodologies for estimating predictive variance networks in regression neural networks. We derive a locally aware mini-batching scheme that result in sparse robust gradients, and show how to make unbiased weight updates to a variance network. Further, we formulate a heuristic for robustly fitting both the mean and variance networks post hoc. Finally, we take inspiration from posterior Gaussian processes and propose a network architecture with similar extrapolation properties to Gaussian processes. The proposed methodologies are complementary, and improve upon baseline methods individually. Experimentally, we investigate the impact on predictive uncertainty on multiple datasets and tasks ranging from regression, active learning and generative modeling. Experiments consistently show significant improvements in predictive uncertainty estimation over state-of-the-art methods across tasks and datasets.