mlsanityData to 14 Oct 2026

Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing

arXiv 2309.12236 ↗First posted 21 Sep 2023cs.LGcited by 63

Not on the map. Why?

Abstract (arXiv)

Calibration measures and reliability diagrams are two fundamental tools for measuring and interpreting the calibration of probabilistic predictors. Calibration measures quantify the degree of miscalibration, and reliability diagrams visualize the structure of this miscalibration. However, the most common constructions of reliability diagrams and calibration measures -- binning and ECE -- both suffer from well-known flaws (e.g. discontinuity). We show that a simple modification fixes both constructions: first smooth the observations using an RBF kernel, then compute the Expected Calibration Error (ECE) of this smoothed function. We prove that with a careful choice of bandwidth, this method yields a calibration measure that is well-behaved in the sense of (Blasiok, Gopalan, Hu, and Nakkiran 2023a) -- a consistent calibration measure. We call this measure the SmoothECE. Moreover, the reliability diagram obtained from this smoothed function visually encodes the SmoothECE, just as binned reliability diagrams encode the BinnedECE. We also provide a Python package with simple, hyperparameter-free methods for measuring and plotting calibration: `pip install relplot`.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

22 of 46 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2019Measuring Calibration in Deep Learningcited by 446
  4. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  5. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  6. 2020Calibration of Pre-trained Transformerscited by 315
  7. 2019Verified Uncertainty Calibrationcited by 309
  8. 2019Evaluating model calibration in classificationcited by 168
  9. 2020Mitigating Bias in Calibration Error Estimationcited by 101
  10. 2020Calibration of Neural Networks using Splinescited by 97
  11. 2022A Unifying Theory of Distance from Calibrationcited by 53
  12. 2022Low-Degree Multicalibrationcited by 45
10 more
  1. 2023When Does Optimizing a Proper Loss Yield Calibration?cited by 36
  2. 2022Metrics of calibration for probabilistic predictionscited by 32
  3. 2020Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisitedcited by 26
  4. 2022T-Cal: An optimal test for the calibration of predictive modelscited by 24
  5. 2022Smooth Calibration, Leaky Forecasts, Finite Recall, and Nash Dynamicscited by 22
  6. 2022Forecast Hedging and Calibrationcited by 22
  7. 2022Calibration tests beyond classificationcited by 20
  8. 2014Binary Classifier Calibration: Non-parametric approachcited by 13
  9. 2023Evaluating Probabilistic Classifiers: The Triptychcited by 8
  10. 2020Plots of the cumulative differences between observed and expected values of ordered Bernoulli variatescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 22; 2026: 28.

63 papers.

Sort:

By area:

  1. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  2. 2024On Verbalized Confidence Scores for LLMscited by 83
  3. 2024Calibrating Large Language Models Using Their Generations Onlycited by 65
  4. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 37
  5. 2024Teaching Large Language Models to Express Knowledge Boundary from Their Own Signalscited by 32
  6. 2024A Survey on the Honesty of Large Language Modelscited by 23
  7. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  8. 2024When is Multicalibration Post-Processing Necessary?cited by 18
  9. 2025Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learningcited by 18
  10. 2025Rethinking Early Stopping: Refine, Then Calibratecited by 17
  11. 2025MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Toolscited by 15
  12. 2023Kandinsky Conformal Prediction: Efficient Calibration of Image Segmentation Algorithmscited by 14
38 more
  1. 2025Can a calibration metric be both testable and actionable?cited by 12
  2. 2025LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilizationcited by 11
  3. 2024Testing Calibration in Nearly-Linear Timecited by 10
  4. 2025Smooth Calibration and Decision Makingcited by 10
  5. 2024Reassessing How to Compare and Improve the Calibration of Machine Learning Modelscited by 8
  6. 2024A Survey of Calibration Process for Black-Box LLMscited by 8
  7. 2025When does a predictor know its own loss?cited by 7
  8. 2025Discretization-free Multicalibration through Loss Minimization over Tree Ensemblescited by 7
  9. 2024Probabilistic Calibration by Design for Neural Network Regressioncited by 6
  10. 2026Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibrationcited by 6
  11. 2025MCGrad: Multicalibration at Web Scalecited by 5
  12. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 5
  13. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 4
  14. 2025Calibration through the Lens of Indistinguishabilitycited by 4
  15. 2025Sample Margin-Aware Recalibration of Temperature Scalingcited by 3
  16. 2025Efficient Calibration for Decision Makingcited by 3
  17. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  18. 2026The Importance of Being Smoothly Calibratedcited by 3
  19. 2025Critical Appraisal of Fairness Metrics in Clinical Predictive AIcited by 2
  20. 2026Uncertainty Quantification for LLM Function-Callingcited by 2
  21. 2026CalArena: A Large-Scale Post-Hoc Calibration Benchmarkcited by 2
  22. 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Modelscited by 2
  23. 2024On the Use of Anchoring for Training Vision Modelscited by 1
  24. 2024Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modelingcited by 1
  25. 2025Probability Bounding: Post-Hoc Calibration via Box-Constrained Softmaxcited by 1
  26. 2025A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysiscited by 1
  27. 2026Confidence Should Be Calibrated More Than One Turn Deepcited by 1
  28. 2026Probing Routing-Conditional Calibration in Attention-Residual Transformerscited by 1
  29. 2026The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trustcited by 1
  30. 2026HakemBench: A Turkish Benchmark of Typed Decisionscited by 1
  31. 2025PostHoc FREE Calibrating on Kolmogorov Arnold Networkscited by 0
  32. 2025Calibration Attention: Learning Reliability-Aware Representations for Vision Transformerscited by 0
  33. 2025Auditability and the Landscape of Distance to Multicalibrationcited by 0
  34. 2025Private Federated Multiclass Post-hoc Calibrationcited by 0
  35. 2026Calibrated Confidence Estimation for Tabular Question Answeringcited by 0
  36. 2026Anatomy of a failure: When, how, and why deep vision fails in scientific domainscited by 0
  37. 2026The Minimax Rate of Perturbed Second-Order Calibrationcited by 0
  38. 2026Discovery of Hidden Miscalibration Regimescited by 0

50 most cited of 63 All 63 (JSON)