# Scalable Utility-Aware Multiclass Calibration

arXiv: https://arxiv.org/abs/2510.25458 · HTML: https://mlsanity.com/paper/2510.25458/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 58 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 30 of 34)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Efficient and Modular Implicit Differentiation](https://mlsanity.com/paper/2105.15183/) (2021)
- [A vector-contraction inequality for Rademacher complexities](https://mlsanity.com/paper/1605.00251/) (2016)
- [Mitigating Bias in Calibration Error Estimation](https://mlsanity.com/paper/2012.08668/) (2020)
- [Calibration of Neural Networks using Splines](https://mlsanity.com/paper/2006.12800/) (2020)
- [Calibration tests in multi-class classification: A unifying framework](https://mlsanity.com/paper/1910.11385/) (2019)
- [Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration](https://mlsanity.com/paper/2107.05719/) (2021)
- [Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks](https://mlsanity.com/paper/2003.06820/) (2020)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [Moment Multicalibration for Uncertainty Estimation](https://mlsanity.com/paper/2008.08037/) (2020)
- [Loss Minimization through the Lens of Outcome Indistinguishability](https://mlsanity.com/paper/2210.08649/) (2022)
- [A Unifying Theory of Distance from Calibration](https://mlsanity.com/paper/2211.16886/) (2022)
- [A Consistent and Differentiable Lp Canonical Calibration Error Estimator](https://mlsanity.com/paper/2210.07810/) (2022)
- [Top-label calibration and multiclass-to-binary reductions](https://mlsanity.com/paper/2107.08353/) (2021)
- [U-Calibration: Forecasting for an Unknown Agent](https://mlsanity.com/paper/2307.00168/) (2023)
- [Distribution-free calibration guarantees for histogram binning without sample splitting](https://mlsanity.com/paper/2105.04656/) (2021)
- [Low-Degree Multicalibration](https://mlsanity.com/paper/2203.01255/) (2022)
- [High-Dimensional Prediction for Sequential Decision Making](https://mlsanity.com/paper/2310.17651/) (2023)
- [Swap Agnostic Learning, or Characterizing Omniprediction via Multicalibration](https://mlsanity.com/paper/2302.06726/) (2023)
- [Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning](https://mlsanity.com/paper/2006.13092/) (2020)
- [Forecasting for Swap Regret for All Downstream Agents](https://mlsanity.com/paper/2402.08753/) (2024)
- [T-Cal: An optimal test for the calibration of predictive models](https://mlsanity.com/paper/2203.01850/) (2022)
- [Calibration by Distribution Matching: Trainable Kernel Calibration Metrics](https://mlsanity.com/paper/2310.20211/) (2023)
- [On Computationally Efficient Multi-Class Calibration](https://mlsanity.com/paper/2402.07821/) (2024)

## Built on by (most cited first, 1 of 1)

- [Noisy-Label Calibration with Multiple Unreliable Labels](https://mlsanity.com/paper/2610.19543/) (2026)

## Abstract (arXiv)

Ensuring that classifiers are well-calibrated, i.e., their predictions align with observed frequencies, is a minimal and fundamental requirement for classifiers to be viewed as trustworthy. Existing methods for assessing multiclass calibration often focus on specific aspects associated with prediction (e.g., top-class confidence, class-wise calibration) or utilize computationally challenging variational formulations. In this work, we study scalable evaluation of multiclass calibration. To this end, we propose utility calibration, a general framework that measures the calibration error relative to a specific utility function that encapsulates the goals or decision criteria relevant to the end user. We demonstrate how this framework can unify and re-interpret several existing calibration metrics, particularly allowing for more robust versions of the top-class and class-wise calibration metrics, and, going beyond such binarized approaches, toward assessing calibration for richer classes of downstream utilities.