# Truthfulness of Decision-Theoretic Calibration Measures

arXiv: https://arxiv.org/abs/2503.02384 · HTML: https://mlsanity.com/paper/2503.02384/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2025.

## Built on by (most cited first, 10 of 10)

- [A Perfectly Truthful Calibration Measure](https://mlsanity.com/paper/2508.13100/) (2025)
- [Panprediction: Optimal Predictions for Any Downstream Task and Loss](https://mlsanity.com/paper/2510.27638/) (2025)
- [Testable and Actionable Calibration for Full Swap Regret](https://mlsanity.com/paper/2605.17749/) (2026)
- [Truthful Calibration Errors for Multi-Class Prediction](https://mlsanity.com/paper/2510.06388/) (2025)
- [Computational and Statistical Hardness of Calibration Distance](https://mlsanity.com/paper/2603.18391/) (2026)
- [Decision-Aligned Evaluation of Uncertainty Quantification](https://mlsanity.com/paper/2606.26990/) (2026)
- [Smoothed Elicitation Complexity for Approximate Gamma-calibration of Discrete Classification Tasks](https://mlsanity.com/paper/2605.23017/) (2026)
- [Calibration without labels in multiple testing](https://mlsanity.com/paper/2606.19737/) (2026)
- [Truthful Calibration Measures for Sequential Prediction](https://mlsanity.com/paper/2608.21348/) (2026)
- [Sharp Integrality Gaps in Calibration Distance](https://mlsanity.com/paper/2610.05679/) (2026)

## Abstract (arXiv)

Calibration measures quantify how much a forecaster's predictions violates calibration, which requires that forecasts are unbiased conditioning on the forecasted probabilities. Two important desiderata for a calibration measure are its decision-theoretic implications (i.e., downstream decision-makers that best-respond to the forecasts are always no-regret) and its truthfulness (i.e., a forecaster approximately minimizes error by always reporting the true probabilities). Existing measures satisfy at most one of the properties, but not both. We introduce a new calibration measure termed subsampled step calibration, mathsfStepCE^textsfsub, that is both decision-theoretic and truthful. In particular, on any product distribution, mathsfStepCE^textsfsub is truthful up to an O(1) factor whereas prior decision-theoretic calibration measures suffer from an e-Omega(T)-Omega(√T) truthfulness gap. Moreover, in any smoothed setting where the conditional probability of each event is perturbed by a noise of magnitude c > 0, mathsfStepCE^textsfsub is truthful up to an O(√log(1/c)) factor, while prior decision-theoretic measures have an e-Omega(T)-Omega(T1/3) truthfulness gap. We also prove a general impossibility result for truthful decision-theoretic forecasting: any complete and decision-theoretic calibration measure must be discontinuous and non-truthful in the non-smoothed setting.