# Practical Deep Heteroskedastic Regression

arXiv: https://arxiv.org/abs/2603.01750 · HTML: https://mlsanity.com/paper/2603.01750/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Builds on (most cited first, 28 of 28)

- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Equivariant message passing for the prediction of tensorial properties and molecular spectra](https://mlsanity.com/paper/2102.03150/) (2021)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Deep Deterministic Uncertainty: A Simple Baseline](https://mlsanity.com/paper/2102.11582/) (2021)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks](https://mlsanity.com/paper/2203.09168/) (2022)
- [UMA: A Family of Universal Models for Atoms](https://mlsanity.com/paper/2506.23971/) (2025)
- [Evaluating and Calibrating Uncertainty Prediction in Regression Tasks](https://mlsanity.com/paper/1905.11659/) (2019)
- [Learning Smooth and Expressive Interatomic Potentials for Physical Property Prediction](https://mlsanity.com/paper/2502.12147/) (2025)
- [Reliable training and estimation of variance networks](https://mlsanity.com/paper/1906.03260/) (2019)
- [Open Materials 2024 (OMat24) Inorganic Materials Dataset and Models](https://mlsanity.com/paper/2410.12771/) (2024)
- [A Study of Bayesian Neural Network Surrogates for Bayesian Optimization](https://mlsanity.com/paper/2305.20028/) (2023)
- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domains](https://mlsanity.com/paper/2410.24169/) (2024)
- [Faithful Heteroscedastic Regression with Neural Networks](https://mlsanity.com/paper/2212.09184/) (2022)
- [Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Models](https://mlsanity.com/paper/1906.09427/) (2019)
- [Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networks](https://mlsanity.com/paper/2107.06068/) (2021)
- [Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks](https://mlsanity.com/paper/2205.10041/) (2022)
- [Large-Scale Visual Active Learning with Deep Probabilistic Ensembles](https://mlsanity.com/paper/1811.03575/) (2018)
- [Bayesian Optimization via Continual Variational Last Layer Training](https://mlsanity.com/paper/2412.09477/) (2024)
- [Understanding Pathologies of Deep Heteroskedastic Regression](https://mlsanity.com/paper/2306.16717/) (2023)
- [Uncertainty Quantification for Molecular Property Predictions with Graph Neural Architecture Search](https://mlsanity.com/paper/2307.10438/) (2023)
- [The Open DAC 2025 Dataset for Sorbent Discovery in Direct Air Capture](https://mlsanity.com/paper/2508.03162/) (2025)
- [Vecchia Gaussian Process Ensembles on Internal Representations of Deep Neural Networks](https://mlsanity.com/paper/2305.17063/) (2023)
- [Probabilistic Skip Connections for Deterministic Uncertainty Quantification in Deep Neural Networks](https://mlsanity.com/paper/2501.04816/) (2025)

## Abstract (arXiv)

Uncertainty quantification (UQ) in deep learning regression is of wide interest, as it supports critical applications including sequential decision making and risk-sensitive tasks. In heteroskedastic regression, where the uncertainty of the target depends on the input, a common approach is to train a neural network that parameterizes the mean and the variance of the predictive distribution. Still, training deep heteroskedastic regression models poses practical challenges in the trade-off between uncertainty quantification and mean prediction, such as optimization difficulties, representation collapse, and variance overfitting. In this work we identify previously undiscussed fallacies and propose a simple and efficient procedure that addresses these challenges jointly by post-hoc fitting a variance model across the intermediate layers of a pretrained network on a hold-out dataset. We demonstrate that our method achieves on-par or state-of-the-art uncertainty quantification on several molecular graph datasets, without compromising mean prediction accuracy and remaining cheap to use at prediction time.