# Calibration by Distribution Matching: Trainable Kernel Calibration Metrics

arXiv: https://arxiv.org/abs/2310.20211 · HTML: https://mlsanity.com/paper/2310.20211/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 31 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [On Fairness and Calibration](https://mlsanity.com/paper/1709.02012/) (2017)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Soft Calibration Objectives for Neural Networks](https://mlsanity.com/paper/2108.00106/) (2021)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Calibration tests in multi-class classification: A unifying framework](https://mlsanity.com/paper/1910.11385/) (2019)
- [Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration](https://mlsanity.com/paper/2107.05719/) (2021)
- [Outcome Indistinguishability](https://mlsanity.com/paper/2011.13426/) (2020)
- [Individual Calibration with Randomized Forecasting](https://mlsanity.com/paper/2006.10288/) (2020)
- [A Consistent and Differentiable Lp Canonical Calibration Error Estimator](https://mlsanity.com/paper/2210.07810/) (2022)
- [Characteristic and Universal Tensor Product Kernels](https://mlsanity.com/paper/1708.08157/) (2017)
- [Distribution Matching in Variational Inference](https://mlsanity.com/paper/1802.06847/) (2018)
- [Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error](https://mlsanity.com/paper/2106.09613/) (2021)
- [Local Calibration: Metrics and Recalibration](https://mlsanity.com/paper/2102.10809/) (2021)
- [Conformal calibrators](https://mlsanity.com/paper/1902.06579/) (2019)
- [Calibration tests beyond classification](https://mlsanity.com/paper/2210.13355/) (2022)
- [Modular Conformal Calibration](https://mlsanity.com/paper/2206.11468/) (2022)

## Built on by (most cited first, 20 of 20)

- [A calibration test for evaluating set-based epistemic uncertainty representations](https://mlsanity.com/paper/2502.16299/) (2025)
- [Calibrated Probabilistic Forecasts for Arbitrary Sequences](https://mlsanity.com/paper/2409.19157/) (2024)
- [Fully Heteroscedastic Count Regression with Deep Double Poisson Networks](https://mlsanity.com/paper/2406.09262/) (2024)
- [Optimizing Estimators of Squared Calibration Errors in Classification](https://mlsanity.com/paper/2410.07014/) (2024)
- [Distilling Calibration via Conformalized Credal Inference](https://mlsanity.com/paper/2501.06066/) (2025)
- [Multivariate Latent Recalibration for Conditional Normalizing Flows](https://mlsanity.com/paper/2505.16636/) (2025)
- [Improving Multi-Class Calibration through Normalization-Aware Isotonic Techniques](https://mlsanity.com/paper/2512.09054/) (2025)
- [Multiclass Local Calibration with the Jensen-Shannon Distance](https://mlsanity.com/paper/2510.26566/) (2025)
- [Retrieval-Augmented Linguistic Calibration](https://mlsanity.com/paper/2605.19344/) (2026)
- [Hierarchies of Calibration: Classification meets Regression](https://mlsanity.com/paper/2606.03245/) (2026)
- [A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessment](https://mlsanity.com/paper/2408.01301/) (2024)
- [I-trustworthy Models. A framework for trustworthiness evaluation of probabilistic classifiers](https://mlsanity.com/paper/2501.15617/) (2025)
- [Scalable Utility-Aware Multiclass Calibration](https://mlsanity.com/paper/2510.25458/) (2025)
- [Uniform convergence of the smooth calibration error and its relationship with functional gradient](https://mlsanity.com/paper/2505.19396/) (2025)
- [Auditability and the Landscape of Distance to Multicalibration](https://mlsanity.com/paper/2509.16930/) (2025)
- [Private Federated Multiclass Post-hoc Calibration](https://mlsanity.com/paper/2510.01987/) (2025)
- [Nonparametric Distribution Regression Re-calibration](https://mlsanity.com/paper/2602.13362/) (2026)
- [Trustworthy AI: Ensuring Reliability and Accountability from Models to Agents](https://mlsanity.com/paper/2605.08964/) (2026)
- [Smooth Learning with Hard Constraints via Legendre-Regularized Policies](https://mlsanity.com/paper/2607.24007/) (2026)
- [SPACE: Sample-cloud Predictive Adaptive Conformal Ellipsoids for Multivariate Time-Series Forecasting](https://mlsanity.com/paper/2608.17333/) (2026)

## Abstract (arXiv)

Calibration ensures that probabilistic forecasts meaningfully capture uncertainty by requiring that predicted probabilities align with empirical frequencies. However, many existing calibration methods are specialized for post-hoc recalibration, which can worsen the sharpness of forecasts. Drawing on the insight that calibration can be viewed as a distribution matching task, we introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. Furthermore, we provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions. Our empirical evaluation demonstrates that employing these metrics as regularizers enhances calibration, sharpness, and decision-making across a range of regression and classification tasks, outperforming methods relying solely on post-hoc recalibration.