mlsanityData to 7 Oct 2026

Distribution-free calibration guarantees for histogram binning without sample splitting

arXiv 2105.04656 ↗First posted 10 May 2021stat.ME cs.LG stat.MLcited by 44

Not on the map. Why?

Abstract (arXiv)

We prove calibration guarantees for the popular histogram binning (also called uniform-mass binning) method of Zadrozny and Elkan [2001]. Histogram binning has displayed strong practical performance, but theoretical guarantees have only been shown for sample split versions that avoid 'double dipping' the data. We demonstrate that the statistical cost of sample splitting is practically significant on a credit default dataset. We then prove calibration guarantees for the original method that double dips the data, using a certain Markov property of order statistics. Based on our results, we make practical recommendations for choosing the number of bins in histogram binning. In our illustrative simulations, we propose a new tool for assessing calibration -- validity plots -- which provide more information than an ECE estimate. Code for this work will be made publicly available at https://github.com/aigen/df-posthoc-calibration.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 6 of its 28 references that are in the map. First posted on arXiv 10 May 2021.

Builds on

6 of 28 references.

References · 6

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2019Verified Uncertainty Calibrationcited by 307
  4. 2020Mitigating Bias in Calibration Error Estimationcited by 100
  5. 2019Calibration tests in multi-class classification: A unifying frameworkcited by 79
  6. 2020Distribution-free binary classification: prediction sets, confidence intervals and calibrationcited by 79

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 7; 2024: 10; 2025: 10; 2026: 7.

44 papers.

Sort:

By area:

  1. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  2. 2021Top-label calibration and multiclass-to-binary reductionscited by 46
  3. 2023Human-Aligned Calibration for AI-Assisted Decision Makingcited by 28
  4. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  5. 2022Improving Screening Processes via Calibrated Subset Selectioncited by 16
  6. 2023Causal isotonic calibration for heterogeneous treatment effectscited by 16
  7. 2024Self-Calibrating Conformal Predictioncited by 16
  8. 2024QA-Calibration of Language Model Confidence Scorescited by 13
  9. 2023Minimum-Risk Recalibration of Classifierscited by 10
  10. 2023A Benchmark Study on Calibrationcited by 10
  11. 2024Information-theoretic Generalization Analysis for Expected Calibration Errorcited by 9
  12. 2021Scaffolding Setscited by 8
32 more
  1. 2022Taking a Step Back with KCal: Multi-Class Kernel-Based Calibration for Deep Neural Networkscited by 8
  2. 2023Online Platt Scaling with Calibeatingcited by 8
  3. 2024Stabilized Inverse Probability Weighting via Isotonic Calibrationcited by 8
  4. 2022Birds of a Feather Trust Together: Knowing When to Trust a Classifier via Adaptive Neighborhood Aggregationcited by 7
  5. 2025Algorithms with Calibrated Machine Learning Predictionscited by 7
  6. 2025Generalized Venn and Venn-Abers Calibration with Applications in Conformal Predictioncited by 6
  7. 2026Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibrationcited by 6
  8. 2024Orthogonal Causal Calibrationcited by 5
  9. 2022Faster online calibration without randomization: interval forecasts and the power of two choicescited by 4
  10. 2022Active Learning and Novel Model Calibration Measurements for Automated Visual Inspection in Manufacturingcited by 4
  11. 2024Improving the Finite Sample Estimation of Average Treatment Effects using Double/Debiased Machine Learning with Propensity Score Calibrationcited by 4
  12. 2024Pearls from Pebbles: Improved Confidence Functions for Auto-labelingcited by 3
  13. 2024PAC-Bayes Analysis for Recalibration in Classificationcited by 3
  14. 2025Efficient Calibration for Decision Makingcited by 3
  15. 2026Calibrating Model-Based Evaluation Metrics for Summarizationcited by 3
  16. 2023Parity Calibrationcited by 2
  17. 2025Bellman Calibration for V-Learning in Offline Reinforcement Learningcited by 2
  18. 2024A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessmentcited by 1
  19. 2025To Trust or Not to Trust: On Calibration in ML-based Resource Allocation for Wireless Networkscited by 1
  20. 2025A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysiscited by 1
  21. 2026Learning to Decide with AI Assistance under Human-Alignmentcited by 1
  22. 2022A Geometric Method for Improved Uncertainty Estimation in Real-timecited by 0
  23. 2022Estimating Classification Confidence Using Kernel Densitiescited by 0
  24. 2023Uncertainty Estimation based on Geometric Separationcited by 0
  25. 2024Sample size planning for conditional counterfactual mean estimation with a K-armed randomized experimentcited by 0
  26. 2025Uniform convergence of the smooth calibration error and its relationship with functional gradientcited by 0
  27. 2025L2-Regularized Empirical Risk Minimization Guarantees Small Smooth Calibration Errorcited by 0
  28. 2025Scalable Utility-Aware Multiclass Calibrationcited by 0
  29. 2026When Models Know When They Do Not Know: Calibration, Cascading, and Cleaningcited by 0
  30. 2026Instance-Adaptive Online Multicalibrationcited by 0
  31. 2026Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibrationcited by 0
  32. 2026Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learningcited by 0