# Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression

arXiv: https://arxiv.org/abs/2502.10587 · HTML: https://mlsanity.com/paper/2502.10587/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020)
- [Reliable training and estimation of variance networks](https://mlsanity.com/paper/1906.03260/) (2019)
- [MonoLoco: Monocular 3D Pedestrian Localization and Uncertainty Estimation](https://mlsanity.com/paper/1906.06059/) (2019)
- [Faithful Heteroscedastic Regression with Neural Networks](https://mlsanity.com/paper/2212.09184/) (2022)
- [Structured Uncertainty Prediction Networks](https://mlsanity.com/paper/1802.07079/) (2018)
- [Multivariate Uncertainty in Deep Learning](https://mlsanity.com/paper/1910.14215/) (2019)
- [Few-shot Keypoint Detection with Uncertainty Learning for Unseen Species](https://mlsanity.com/paper/2112.06183/) (2021)
- [On the Properties of Kullback-Leibler Divergence Between Multivariate Gaussian Distributions](https://mlsanity.com/paper/2102.05485/) (2021)
- [Optimal Training of Mean Variance Estimation Neural Networks](https://mlsanity.com/paper/2302.08875/) (2023)
- [Localization with Sampling-Argmax](https://mlsanity.com/paper/2110.08825/) (2021)
- [Understanding Pathologies of Deep Heteroskedastic Regression](https://mlsanity.com/paper/2306.16717/) (2023)
- [Learning Structured Gaussians to Approximate Deep Ensembles](https://mlsanity.com/paper/2203.15485/) (2022)
- [TIC-TAC: A Framework for Improved Covariance Estimation in Deep Heteroscedastic Regression](https://mlsanity.com/paper/2310.18953/) (2023)
- [Understanding Pose and Appearance Disentanglement in 3D Human Pose Estimation](https://mlsanity.com/paper/2309.11667/) (2023)

## Built on by (most cited first, 2 of 2)

- [UPLME: Uncertainty-Aware Probabilistic Language Modelling for Robust Empathy Regression](https://mlsanity.com/paper/2508.03520/) (2025)
- [On the Effect of Regularization on Nonparametric Mean-Variance Regression](https://mlsanity.com/paper/2511.22004/) (2025)

## Abstract (arXiv)

Deep heteroscedastic regression models the mean and covariance of the target distribution through neural networks. The challenge arises from heteroscedasticity, which implies that the covariance is sample dependent and is often unknown. Consequently, recent methods learn the covariance through unsupervised frameworks, which unfortunately yield a trade-off between computational complexity and accuracy. While this trade-off could be alleviated through supervision, obtaining labels for the covariance is non-trivial. Here, we study self-supervised covariance estimation in deep heteroscedastic regression. We address two questions: (1) How should we supervise the covariance assuming ground truth is available? (2) How can we obtain pseudo labels in the absence of the ground-truth? We address (1) by analysing two popular measures: the KL Divergence and the 2-Wasserstein distance. Subsequently, we derive an upper bound on the 2-Wasserstein distance between normal distributions with non-commutative covariances that is stable to optimize. We address (2) through a simple neighborhood based heuristic algorithm which results in surprisingly effective pseudo labels for the covariance. Our experiments over a wide range of synthetic and real datasets demonstrate that the proposed 2-Wasserstein bound coupled with pseudo label annotations results in a computationally cheaper yet accurate deep heteroscedastic regression.