# On the Effect of Regularization on Nonparametric Mean-Variance Regression

arXiv: https://arxiv.org/abs/2511.22004 · HTML: https://mlsanity.com/paper/2511.22004/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 83 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Builds on (most cited first, 27 of 27)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Statistical physics of inference: Thresholds and algorithms](https://mlsanity.com/paper/1511.02476/) (2015)
- [Evaluating and Calibrating Uncertainty Prediction in Regression Tasks](https://mlsanity.com/paper/1905.11659/) (2019)
- [Reliable training and estimation of variance networks](https://mlsanity.com/paper/1906.03260/) (2019)
- [The jamming transition as a paradigm to understand the loss landscape of deep neural networks](https://mlsanity.com/paper/1809.09349/) (2018)
- [Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks](https://mlsanity.com/paper/2202.00293/) (2022)
- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [Faithful Heteroscedastic Regression with Neural Networks](https://mlsanity.com/paper/2212.09184/) (2022)
- [Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitions](https://mlsanity.com/paper/1804.02686/) (2018)
- [Variational Tempering](https://mlsanity.com/paper/1411.1810/) (2014)
- [Optimal Training of Mean Variance Estimation Neural Networks](https://mlsanity.com/paper/2302.08875/) (2023)
- [The SPDE approach for Gaussian and non-Gaussian fields: 10 years and still running](https://mlsanity.com/paper/2111.01084/) (2021)
- [Understanding Pathologies of Deep Heteroskedastic Regression](https://mlsanity.com/paper/2306.16717/) (2023)
- [Glassy nature of the hard phase in inference problems](https://mlsanity.com/paper/1805.05857/) (2018)
- [Applications of Statistical Field Theory in Deep Learning](https://mlsanity.com/paper/2502.18553/) (2025)
- [Improving Optimization for Models With Continuous Symmetry Breaking](https://mlsanity.com/paper/1803.03234/) (2018)
- [Deep Classifiers with Label Noise Modeling and Distance Awareness](https://mlsanity.com/paper/2110.02609/) (2021)
- [Training shallow ReLU networks on noisy data using hinge loss: when do we overfit and is it benign?](https://mlsanity.com/paper/2306.09955/) (2023)
- [Neural Active Learning on Heteroskedastic Distributions](https://mlsanity.com/paper/2211.00928/) (2022)
- [TIC-TAC: A Framework for Improved Covariance Estimation in Deep Heteroscedastic Regression](https://mlsanity.com/paper/2310.18953/) (2023)
- [Deep Generative Models: Complexity, Dimensionality, and Approximation](https://mlsanity.com/paper/2504.00820/) (2025)
- [Precise Asymptotic Generalization for Multiclass Classification with Overparameterized Linear Models](https://mlsanity.com/paper/2306.13255/) (2023)
- [Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression](https://mlsanity.com/paper/2502.10587/) (2025)

## Abstract (arXiv)

Uncertainty quantification is vital for decision-making and risk assessment in machine learning. Mean-variance regression models, which predict both a mean and residual noise for each data point, provide a simple approach to uncertainty quantification. However, overparameterized mean-variance models struggle with signal-to-noise ambiguity, deciding whether prediction targets should be attributed to signal (mean) or noise (variance). At one extreme, models fit all training targets perfectly with zero residual noise, while at the other, they provide constant, uninformative predictions and explain the targets as noise. We observe a sharp phase transition between these extremes, driven by model regularization. Empirical studies with varying regularization levels illustrate this transition, revealing substantial variability across repeated runs. To explain this behavior, we develop a statistical field theory framework, which captures the observed phase transition in alignment with experimental results. This analysis reduces the regularization hyperparameter search space from two dimensions to one, significantly lowering computational costs. Experiments on UCI datasets and the large-scale ClimSim dataset demonstrate robust calibration performance, effectively quantifying predictive uncertainty.