mlsanityData to 7 Oct 2026

Optimal Training of Mean Variance Estimation Neural Networks

arXiv 2302.08875 ↗First posted 17 Feb 2023stat.ML cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

This paper focusses on the optimal implementation of a Mean Variance Estimation network (MVE network) (Nix and Weigend, 1994). This type of network is often used as a building block for uncertainty estimation methods in a regression setting, for instance Concrete dropout (Gal et al., 2017) and Deep Ensembles (Lakshminarayanan et al., 2017). Specifically, an MVE network assumes that the data is produced from a normal distribution with a mean function and variance function. The MVE network outputs a mean and variance estimate and optimizes the network parameters by minimizing the negative loglikelihood. In our paper, we present two significant insights. Firstly, the convergence difficulties reported in recent work can be relatively easily prevented by following the simple yet often overlooked recommendation from the original authors that a warm-up period should be used. During this period, only the mean is optimized with a fixed variance. We demonstrate the effectiveness of this step through experimentation, highlighting that it should be standard practice. As a sidenote, we examine whether, after the warm-up, it is beneficial to fix the mean while optimizing the variance or to optimize both simultaneously. Here, we do not observe a substantial difference. Secondly, we introduce a novel improvement of the MVE network: separate regularization of the mean and the variance estimate. We demonstrate, both on toy examples and on a number of benchmark UCI regression data sets, that following the original recommendations and the novel separate regularization can lead to significant improvements.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 13 of its 33 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

13 of 33 references.

References · 13

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  5. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  6. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  7. 2017Concrete Dropoutcited by 227
  8. 2022On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networkscited by 119
  9. 2018High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approachcited by 117
  10. 2019Reliable training and estimation of variance networkscited by 63
  11. 2021AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantificationcited by 31
  12. 2019Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensemblescited by 30
1 more
  1. 2015Lecture notes on ridge regressioncited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 8; 2026: 6.

17 papers.

Sort:

  1. 2025Bias-variance decompositions: the exclusive privilege of Bregman divergencescited by 6
  2. 2026Learning Credal Ensembles via Distributionally Robust Optimizationcited by 6
  3. 2023Identifying Drivers of Predictive Aleatoric Uncertaintycited by 4
  4. 2024Neural Network Prediction of Strong Lensing Systems with Domain Adaptation and Uncertainty Quantificationcited by 2
  5. 2025Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regressioncited by 2
  6. 2025Probabilistic Neural Networks (PNNs) with t-Distributed Outputs: Adaptive Prediction Intervals Beyond Gaussian Assumptionscited by 1
  7. 2025Improving Counterfactual Truthfulness for Molecular Property Prediction through Uncertainty Quantificationcited by 1
  8. 2025IAEmu: Learning Galaxy Intrinsic Alignment Correlationscited by 1
  9. 2025Cooperative Variance Estimation and Bayesian Neural Networks for Disentangling Aleatoric and Epistemic Uncertaintiescited by 1
  10. 2026Optimized Certainty Equivalent Risk Minimization Using Samples: Algorithms, Convergence Rates, and Applicationscited by 1
  11. 2024On the power of data augmentation for head pose estimationcited by 0
  12. 2025BATIS: Bayesian Approaches for Targeted Improvement of Species Distribution Modelscited by 0
5 more
  1. 2025On the Effect of Regularization on Nonparametric Mean-Variance Regressioncited by 0
  2. 2026Partial Effective Information Decomposition for Synergistic Causalitycited by 0
  3. 2026Conformalised imprecise inference for robust extrapolation under limited datacited by 0
  4. 2026TSCoNet: A Two-Stage Copula CNN-LSTM for Uncertainty-Aware Spatio-Temporal Forecastingcited by 0
  5. 2026Two-Step MV-DeepONet: Probabilistic Operator Learning for Uncertainty Propagation Driven by Random Input Fieldscited by 0