# Faithful Heteroscedastic Regression with Neural Networks

arXiv: https://arxiv.org/abs/2212.09184 · HTML: https://mlsanity.com/paper/2212.09184/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 18 Dec 2022.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks](https://mlsanity.com/paper/2203.09168/) (2022)
- [Reliable training and estimation of variance networks](https://mlsanity.com/paper/1906.03260/) (2019)
- [Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization](https://mlsanity.com/paper/2006.04910/) (2020)

## Built on by (most cited first, 30 of 37)

- [UnCRtainTS: Uncertainty Quantification for Cloud Removal in Optical Satellite Time Series](https://mlsanity.com/paper/2304.05464/) (2023)
- [Coherent Soft Imitation Learning](https://mlsanity.com/paper/2305.16498/) (2023)
- [Understanding Pathologies of Deep Heteroskedastic Regression](https://mlsanity.com/paper/2306.16717/) (2023)
- [End-to-End Probabilistic Framework for Learning with Hard Constraints](https://mlsanity.com/paper/2506.07003/) (2025)
- [Robust Estimation of Causal Heteroscedastic Noise Models](https://mlsanity.com/paper/2312.10102/) (2023)
- [TIC-TAC: A Framework for Improved Covariance Estimation in Deep Heteroscedastic Regression](https://mlsanity.com/paper/2310.18953/) (2023)
- [Likelihood Annealing: Fast Calibrated Uncertainty for Regression](https://mlsanity.com/paper/2302.11012/) (2023)
- [Fully Heteroscedastic Count Regression with Deep Double Poisson Networks](https://mlsanity.com/paper/2406.09262/) (2024)
- [HybridFlow: Quantification of Aleatoric and Epistemic Uncertainty with a Single Hybrid Model](https://mlsanity.com/paper/2510.05054/) (2025)
- [Probabilistic Neural Networks (PNNs) for Modeling Aleatoric Uncertainty in Scientific Machine Learning](https://mlsanity.com/paper/2402.13945/) (2024)
- [FairlyUncertain: A Comprehensive Benchmark of Uncertainty in Algorithmic Fairness](https://mlsanity.com/paper/2410.02005/) (2024)
- [Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression](https://mlsanity.com/paper/2502.10587/) (2025)
- [CLAPS: Aleatoric-Epistemic Scaling via Last-Layer Laplace for Conformal Regression](https://mlsanity.com/paper/2512.01384/) (2025)
- [PARIC: Probabilistic Attention Regularization for Language Guided Image Classification from Pre-trained Vison Language Models](https://mlsanity.com/paper/2503.11360/) (2025)
- [Probabilistic Neural Networks (PNNs) with t-Distributed Outputs: Adaptive Prediction Intervals Beyond Gaussian Assumptions](https://mlsanity.com/paper/2503.12354/) (2025)
- [FruitProm: Probabilistic Maturity Estimation and Detection of Fruits and Vegetables](https://mlsanity.com/paper/2510.24885/) (2025)
- [A Tale of Two Variances: When Single-Seed Benchmarks Fail in Bayesian Deep Learning](https://mlsanity.com/paper/2604.23114/) (2026)
- [XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policies](https://mlsanity.com/paper/2605.10734/) (2026)
- [Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence](https://mlsanity.com/paper/2405.12412/) (2024)
- [LucidAtlas: Learning Uncertainty-Aware, Covariate-Disentangled, Individualized Atlas Representations](https://mlsanity.com/paper/2502.08445/) (2025)
- [Uncertainty-Aware Remaining Lifespan Prediction from Images](https://mlsanity.com/paper/2506.13430/) (2025)
- [On the Effect of Regularization on Nonparametric Mean-Variance Regression](https://mlsanity.com/paper/2511.22004/) (2025)
- [Estimating the Event-Related Potential from Few EEG Trials](https://mlsanity.com/paper/2511.23162/) (2025)
- [ImmuVis: Hyperconvolutional Foundation Model for Imaging Mass Cytometry](https://mlsanity.com/paper/2602.04585/) (2026)
- [Moment Matters: Mean and Variance Causal Graph Discovery from Heteroscedastic Observational Data](https://mlsanity.com/paper/2602.23602/) (2026)
- [Practical Deep Heteroskedastic Regression](https://mlsanity.com/paper/2603.01750/) (2026)
- [Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding](https://mlsanity.com/paper/2605.08446/) (2026)
- [Variance-aware Reward Modeling with Anchor Guidance](https://mlsanity.com/paper/2605.11865/) (2026)
- [Joint Model and Data Sparsification via the Marginal Likelihood](https://mlsanity.com/paper/2605.29908/) (2026)
- [FlowSDR: Sufficient Dimension Reduction via Conditional Normalizing Flows](https://mlsanity.com/paper/2606.01346/) (2026)

## Abstract (arXiv)

Heteroscedastic regression models a Gaussian variable's mean and variance as a function of covariates. Parametric methods that employ neural networks for these parameter maps can capture complex relationships in the data. Yet, optimizing network parameters via log likelihood gradients can yield suboptimal mean and uncalibrated variance estimates. Current solutions side-step this optimization problem with surrogate objectives or Bayesian treatments. Instead, we make two simple modifications to optimization. Notably, their combination produces a heteroscedastic model with mean estimates that are provably as accurate as those from its homoscedastic counterpart (i.e.~fitting the mean under squared error loss). For a wide variety of network and task complexities, we find that mean estimates from existing heteroscedastic solutions can be significantly less accurate than those from an equivalently expressive mean-only model. Our approach provably retains the accuracy of an equally flexible mean-only model while also offering best-in-class variance calibration. Lastly, we show how to leverage our method to recover the underlying heteroscedastic noise variance.