# On the Richness of Calibration

arXiv: https://arxiv.org/abs/2302.04118 · HTML: https://mlsanity.com/paper/2302.04118/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 52 references that are in the map.
- First posted on arXiv 8 Feb 2023.

## Builds on (most cited first, 17 of 17)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [Multiaccuracy: Black-Box Post-Processing for Fairness in Classification](https://mlsanity.com/paper/1805.12317/) (2018)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Algorithmic Monoculture and Social Welfare](https://mlsanity.com/paper/2101.05853/) (2021)
- [A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices](https://mlsanity.com/paper/1807.00787/) (2018)
- [Fairness risk measures](https://mlsanity.com/paper/1901.08665/) (2019)
- [Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation](https://mlsanity.com/paper/2205.04610/) (2022)
- [Low-Degree Multicalibration](https://mlsanity.com/paper/2203.01255/) (2022)
- [Local Calibration: Metrics and Recalibration](https://mlsanity.com/paper/2102.10809/) (2021)
- [Risk Measures and Upper Probabilities: Coherence and Stratification](https://mlsanity.com/paper/2206.03183/) (2022)
- [Scaffolding Sets](https://mlsanity.com/paper/2111.03135/) (2021)
- [Escaping the Impossibility of Fairness: From Formal to Substantive Algorithmic Fairness](https://mlsanity.com/paper/2107.04642/) (2021)
- [Variable-Based Calibration for Machine Learning Classifiers](https://mlsanity.com/paper/2209.15154/) (2022)

## Built on by (most cited first, 9 of 9)

- [Scoring Rules and Calibration for Imprecise Probabilities](https://mlsanity.com/paper/2410.23001/) (2024)
- [Three Types of Calibration with Properties and their Semantic and Formal Relationships](https://mlsanity.com/paper/2504.18395/) (2025)
- [Insights From Insurance for Fair Machine Learning](https://mlsanity.com/paper/2306.14624/) (2023)
- [Geospatial Disparities: A Case Study on Real Estate Prices in Paris](https://mlsanity.com/paper/2401.16197/) (2024)
- [On Calibration in Multi-Distribution Learning](https://mlsanity.com/paper/2412.14142/) (2024)
- [Does calibration mean what they say it means; or, the reference class problem rises again](https://mlsanity.com/paper/2412.16769/) (2024)
- [I-trustworthy Models. A framework for trustworthiness evaluation of probabilistic classifiers](https://mlsanity.com/paper/2501.15617/) (2025)
- [Human-AI Teaming Through the Lens of Calibration](https://mlsanity.com/paper/2606.10906/) (2026)
- [A Unifying Perspective on Probabilities as Model Predictions](https://mlsanity.com/paper/2609.09855/) (2026)

## Abstract (arXiv)

Probabilistic predictions can be evaluated through comparisons with observed label frequencies, that is, through the lens of calibration. Recent scholarship on algorithmic fairness has started to look at a growing variety of calibration-based objectives under the name of multi-calibration but has still remained fairly restricted. In this paper, we explore and analyse forms of evaluation through calibration by making explicit the choices involved in designing calibration scores. We organise these into three grouping choices and a choice concerning the agglomeration of group errors. This provides a framework for comparing previously proposed calibration scores and helps to formulate novel ones with desirable mathematical properties. In particular, we explore the possibility of grouping datapoints based on their input features rather than on predictions and formally demonstrate advantages of such approaches. We also characterise the space of suitable agglomeration functions for group errors, generalising previously proposed calibration scores. Complementary to such population-level scores, we explore calibration scores at the individual level and analyse their relationship to choices of grouping. We draw on these insights to introduce and axiomatise fairness deviation measures for population-level scores. We demonstrate that with appropriate choices of grouping, these novel global fairness scores can provide notions of (sub-)group or individual fairness.