mlsanityData to 7 Oct 2026

A Large-Scale Study of Probabilistic Calibration in Neural Network Regression

arXiv 2306.02738 ↗First posted 5 Jun 2023cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Accurate probabilistic predictions are essential for optimal decision making. While neural network miscalibration has been studied primarily in classification, we investigate this in the less-explored domain of regression. We conduct the largest empirical study to date to assess the probabilistic calibration of neural networks. We also analyze the performance of recalibration, conformal, and regularization methods to enhance probabilistic calibration. Additionally, we introduce novel differentiable recalibration and regularization methods, uncovering new insights into their effectiveness. Our findings reveal that regularization methods offer a favorable tradeoff between calibration and sharpness. Post-hoc methods exhibit superior probabilistic calibration, which we attribute to the finite-sample coverage guarantee of conformal prediction. Furthermore, we demonstrate that quantile recalibration can be considered as a specific case of conformal prediction. Our study is fully reproducible and implemented in a common code base for fair comparisons.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

34 of 50 references.

References · 34

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  5. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  6. 2019Conformalized Quantile Regressioncited by 620
  7. 2018Deep learning for time series classification: a reviewcited by 575
  8. 20193D Packing for Self-Supervised Monocular Depth Estimationcited by 437
  9. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  10. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  11. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 336
  12. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
22 more
  1. 2020Fast Differentiable Sorting and Rankingcited by 196
  2. 2018High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approachcited by 117
  3. 2015Should we really use post-hoc tests based on mean-ranks?cited by 116
  4. 2019Differentiable Ranks and Sorting using Optimal Transportcited by 113
  5. 2019Stochastic Optimization of Sorting Networks via Continuous Relaxationscited by 111
  6. 2019Distributional conformal predictioncited by 104
  7. 2021Soft Calibration Objectives for Neural Networkscited by 89
  8. 2019Distribution Calibration for Regressioncited by 88
  9. 2019An Open Source AutoML Benchmarkcited by 85
  10. 2020Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantificationcited by 77
  11. 2019Flexible distribution-free conditional predictive bands using density estimatorscited by 64
  12. 2020Individual Calibration with Randomized Forecastingcited by 50
  13. 2021Improving Conditional Coverage via Orthogonal Quantile Regressioncited by 48
  14. 2022A Consistent and Differentiable Lp Canonical Calibration Error Estimatorcited by 44
  15. 2021Calibrated and Sharp Uncertainties in Deep Learning via Density Estimationcited by 33
  16. 2021Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determinationcited by 31
  17. 2019Conformal calibratorscited by 22
  18. 2021Flexible Model Aggregation for Quantile Regressioncited by 22
  19. 2023ESD: Expected Squared Difference as a Tuning-Free Trainable Calibration Measurecited by 13
  20. 2020Estimating Uncertainty Intervals from Collaborating Networkscited by 10
  21. 2020Quantile Regularization: Towards Implicit Calibration of Regression Modelscited by 8
  22. 2020Designing Accurate Emulators for Scientific Processes using Calibration-Driven Deep Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 9; 2026: 6.

26 papers.

Sort:

By area:

  1. 2023L-C2ST: Local Diagnostics for Posterior Approximations in Simulation-Based Inferencecited by 17
  2. 2025Rectifying Conformity Scores for Better Conditional Coveragecited by 17
  3. 2024Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shiftscited by 7
  4. 2024Probabilistic Calibration by Design for Neural Network Regressioncited by 6
  5. 2026ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rulescited by 4
  6. 2024Fully Heteroscedastic Count Regression with Deep Double Poisson Networkscited by 3
  7. 2025Multivariate Latent Recalibration for Conditional Normalizing Flowscited by 3
  8. 2024Achieving Well-Informed Decision-Making in Drug Discovery: A Comprehensive Calibration Study using Neural Network-Based Structure-Activity Modelscited by 2
  9. 2025Knowing When to Quit: Probabilistic Early Exits for Speech Separationcited by 2
  10. 2025Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?cited by 2
  11. 2026Calibrated Multivariate Distributional Regression with Pre-Rank Regularizationcited by 2
  12. 2024Model-Free Local Recalibration of Neural Networkscited by 1
14 more
  1. 2024A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessmentcited by 1
  2. 2025Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Modelscited by 1
  3. 2025On Equivariant Model Selection through the Lens of Uncertaintycited by 1
  4. 2025Evaluating the Quality of the Quantified Uncertainty for (Re)Calibration of Data-Driven Regression Modelscited by 1
  5. 2026Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Modelscited by 1
  6. 2026Hierarchies of Calibration: Classification meets Regressioncited by 1
  7. 2024Forecasting VIX using Bayesian Deep Learningcited by 0
  8. 2024Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruencecited by 0
  9. 2024Calibrated Multivariate Regression with Localized PIT Mappingscited by 0
  10. 2024Preventing Conflicting Gradients in Neural Marked Temporal Point Processescited by 0
  11. 2025L2-Regularized Empirical Risk Minimization Guarantees Small Smooth Calibration Errorcited by 0
  12. 2025Enforcing Calibration in Multi-Output Probabilistic Regression with Pre-rank Regularizationcited by 0
  13. 2026VOLTA: The Surprising Ineffectiveness of Auxiliary Losses for Calibrated Deep Learningcited by 0
  14. 2026Conservative neural posterior estimation via distributionally robust trainingcited by 0