# A Large-Scale Study of Probabilistic Calibration in Neural Network Regression

arXiv: https://arxiv.org/abs/2306.02738 · HTML: https://mlsanity.com/paper/2306.02738/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 30 of 34)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [Fast Differentiable Sorting and Ranking](https://mlsanity.com/paper/2002.08871/) (2020)
- [High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach](https://mlsanity.com/paper/1802.07167/) (2018)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [Differentiable Ranks and Sorting using Optimal Transport](https://mlsanity.com/paper/1905.11885/) (2019)
- [Stochastic Optimization of Sorting Networks via Continuous Relaxations](https://mlsanity.com/paper/1903.08850/) (2019)
- [Distributional conformal prediction](https://mlsanity.com/paper/1909.07889/) (2019)
- [Soft Calibration Objectives for Neural Networks](https://mlsanity.com/paper/2108.00106/) (2021)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification](https://mlsanity.com/paper/2011.09588/) (2020)
- [Flexible distribution-free conditional predictive bands using density estimators](https://mlsanity.com/paper/1910.05575/) (2019)
- [Individual Calibration with Randomized Forecasting](https://mlsanity.com/paper/2006.10288/) (2020)
- [Improving Conditional Coverage via Orthogonal Quantile Regression](https://mlsanity.com/paper/2106.00394/) (2021)
- [A Consistent and Differentiable Lp Canonical Calibration Error Estimator](https://mlsanity.com/paper/2210.07810/) (2022)
- [Calibrated and Sharp Uncertainties in Deep Learning via Density Estimation](https://mlsanity.com/paper/2112.07184/) (2021)
- [Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination](https://mlsanity.com/paper/2108.03210/) (2021)
- [Conformal calibrators](https://mlsanity.com/paper/1902.06579/) (2019)
- [Flexible Model Aggregation for Quantile Regression](https://mlsanity.com/paper/2103.00083/) (2021)

## Built on by (most cited first, 27 of 27)

- [L-C2ST: Local Diagnostics for Posterior Approximations in Simulation-Based Inference](https://mlsanity.com/paper/2306.03580/) (2023)
- [Rectifying Conformity Scores for Better Conditional Coverage](https://mlsanity.com/paper/2502.16336/) (2025)
- [Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shifts](https://mlsanity.com/paper/2403.05600/) (2024)
- [Probabilistic Calibration by Design for Neural Network Regression](https://mlsanity.com/paper/2403.11964/) (2024)
- [ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules](https://mlsanity.com/paper/2603.29928/) (2026)
- [Fully Heteroscedastic Count Regression with Deep Double Poisson Networks](https://mlsanity.com/paper/2406.09262/) (2024)
- [Multivariate Latent Recalibration for Conditional Normalizing Flows](https://mlsanity.com/paper/2505.16636/) (2025)
- [Achieving Well-Informed Decision-Making in Drug Discovery: A Comprehensive Calibration Study using Neural Network-Based Structure-Activity Models](https://mlsanity.com/paper/2407.14185/) (2024)
- [Knowing When to Quit: Probabilistic Early Exits for Speech Separation](https://mlsanity.com/paper/2507.09768/) (2025)
- [Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?](https://mlsanity.com/paper/2510.16060/) (2025)
- [Calibrated Multivariate Distributional Regression with Pre-Rank Regularization](https://mlsanity.com/paper/2601.22895/) (2026)
- [Hierarchies of Calibration: Classification meets Regression](https://mlsanity.com/paper/2606.03245/) (2026)
- [Model-Free Local Recalibration of Neural Networks](https://mlsanity.com/paper/2403.05756/) (2024)
- [A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessment](https://mlsanity.com/paper/2408.01301/) (2024)
- [Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Models](https://mlsanity.com/paper/2502.03982/) (2025)
- [On Equivariant Model Selection through the Lens of Uncertainty](https://mlsanity.com/paper/2506.18629/) (2025)
- [Evaluating the Quality of the Quantified Uncertainty for (Re)Calibration of Data-Driven Regression Models](https://mlsanity.com/paper/2508.17761/) (2025)
- [Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models](https://mlsanity.com/paper/2604.20472/) (2026)
- [Forecasting VIX using Bayesian Deep Learning](https://mlsanity.com/paper/2401.17042/) (2024)
- [Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence](https://mlsanity.com/paper/2405.12412/) (2024)
- [Calibrated Multivariate Regression with Localized PIT Mappings](https://mlsanity.com/paper/2409.10855/) (2024)
- [Preventing Conflicting Gradients in Neural Marked Temporal Point Processes](https://mlsanity.com/paper/2412.08590/) (2024)
- [L2-Regularized Empirical Risk Minimization Guarantees Small Smooth Calibration Error](https://mlsanity.com/paper/2510.13450/) (2025)
- [Enforcing Calibration in Multi-Output Probabilistic Regression with Pre-rank Regularization](https://mlsanity.com/paper/2510.21273/) (2025)
- [VOLTA: The Surprising Ineffectiveness of Auxiliary Losses for Calibrated Deep Learning](https://mlsanity.com/paper/2604.08639/) (2026)
- [Conservative neural posterior estimation via distributionally robust training](https://mlsanity.com/paper/2605.28516/) (2026)
- [How corner is a corner case? Percentile control for highway scenario generation](https://mlsanity.com/paper/2610.05003/) (2026)

## Abstract (arXiv)

Accurate probabilistic predictions are essential for optimal decision making. While neural network miscalibration has been studied primarily in classification, we investigate this in the less-explored domain of regression. We conduct the largest empirical study to date to assess the probabilistic calibration of neural networks. We also analyze the performance of recalibration, conformal, and regularization methods to enhance probabilistic calibration. Additionally, we introduce novel differentiable recalibration and regularization methods, uncovering new insights into their effectiveness. Our findings reveal that regularization methods offer a favorable tradeoff between calibration and sharpness. Post-hoc methods exhibit superior probabilistic calibration, which we attribute to the finite-sample coverage guarantee of conformal prediction. Furthermore, we demonstrate that quantile recalibration can be considered as a specific case of conformal prediction. Our study is fully reproducible and implemented in a common code base for fair comparisons.