# Optimal Training of Mean Variance Estimation Neural Networks

arXiv: https://arxiv.org/abs/2302.08875 · HTML: https://mlsanity.com/paper/2302.08875/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 17 Feb 2023.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks](https://mlsanity.com/paper/2203.09168/) (2022)
- [High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach](https://mlsanity.com/paper/1802.07167/) (2018)
- [Reliable training and estimation of variance networks](https://mlsanity.com/paper/1906.03260/) (2019)
- [AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification](https://mlsanity.com/paper/2110.13511/) (2021)
- [Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles](https://mlsanity.com/paper/1906.07380/) (2019)
- [Lecture notes on ridge regression](https://mlsanity.com/paper/1509.09169/) (2015)

## Built on by (most cited first, 18 of 18)

- [Learning Credal Ensembles via Distributionally Robust Optimization](https://mlsanity.com/paper/2602.08470/) (2026)
- [Bias-variance decompositions: the exclusive privilege of Bregman divergences](https://mlsanity.com/paper/2501.18581/) (2025)
- [Identifying Drivers of Predictive Aleatoric Uncertainty](https://mlsanity.com/paper/2312.07252/) (2023)
- [Neural Network Prediction of Strong Lensing Systems with Domain Adaptation and Uncertainty Quantification](https://mlsanity.com/paper/2411.03334/) (2024)
- [Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression](https://mlsanity.com/paper/2502.10587/) (2025)
- [CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization](https://mlsanity.com/paper/2602.19699/) (2026)
- [Probabilistic Neural Networks (PNNs) with t-Distributed Outputs: Adaptive Prediction Intervals Beyond Gaussian Assumptions](https://mlsanity.com/paper/2503.12354/) (2025)
- [Improving Counterfactual Truthfulness for Molecular Property Prediction through Uncertainty Quantification](https://mlsanity.com/paper/2504.02606/) (2025)
- [IAEmu: Learning Galaxy Intrinsic Alignment Correlations](https://mlsanity.com/paper/2504.05235/) (2025)
- [Cooperative Variance Estimation and Bayesian Neural Networks for Disentangling Aleatoric and Epistemic Uncertainties](https://mlsanity.com/paper/2505.02743/) (2025)
- [Optimized Certainty Equivalent Risk Minimization Using Samples: Algorithms, Convergence Rates, and Applications](https://mlsanity.com/paper/2608.07113/) (2026)
- [On the power of data augmentation for head pose estimation](https://mlsanity.com/paper/2407.05357/) (2024)
- [BATIS: Bayesian Approaches for Targeted Improvement of Species Distribution Models](https://mlsanity.com/paper/2510.19749/) (2025)
- [On the Effect of Regularization on Nonparametric Mean-Variance Regression](https://mlsanity.com/paper/2511.22004/) (2025)
- [Partial Effective Information Decomposition for Synergistic Causality](https://mlsanity.com/paper/2605.03267/) (2026)
- [Conformalised imprecise inference for robust extrapolation under limited data](https://mlsanity.com/paper/2605.25882/) (2026)
- [TSCoNet: A Two-Stage Copula CNN-LSTM for Uncertainty-Aware Spatio-Temporal Forecasting](https://mlsanity.com/paper/2607.10410/) (2026)
- [Two-Step MV-DeepONet: Probabilistic Operator Learning for Uncertainty Propagation Driven by Random Input Fields](https://mlsanity.com/paper/2608.09071/) (2026)

## Abstract (arXiv)

This paper focusses on the optimal implementation of a Mean Variance Estimation network (MVE network) (Nix and Weigend, 1994). This type of network is often used as a building block for uncertainty estimation methods in a regression setting, for instance Concrete dropout (Gal et al., 2017) and Deep Ensembles (Lakshminarayanan et al., 2017). Specifically, an MVE network assumes that the data is produced from a normal distribution with a mean function and variance function. The MVE network outputs a mean and variance estimate and optimizes the network parameters by minimizing the negative loglikelihood. In our paper, we present two significant insights. Firstly, the convergence difficulties reported in recent work can be relatively easily prevented by following the simple yet often overlooked recommendation from the original authors that a warm-up period should be used. During this period, only the mean is optimized with a fixed variance. We demonstrate the effectiveness of this step through experimentation, highlighting that it should be standard practice. As a sidenote, we examine whether, after the warm-up, it is beneficial to fix the mean while optimizing the variance or to optimize both simultaneously. Here, we do not observe a substantial difference. Secondly, we introduce a novel improvement of the MVE network: separate regularization of the mean and the variance estimate. We demonstrate, both on toy examples and on a number of benchmark UCI regression data sets, that following the original recommendations and the novel separate regularization can lead to significant improvements.