# Calibration for the (Computationally-Identifiable) Masses

arXiv: https://arxiv.org/abs/1711.08513 · HTML: https://mlsanity.com/paper/1711.08513/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2017.

## Built on by (most cited first, 23 of 23)

- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification](https://mlsanity.com/paper/2109.10254/) (2021)
- [Low-Degree Multicalibration](https://mlsanity.com/paper/2203.01255/) (2022)
- [Local Calibration: Metrics and Recalibration](https://mlsanity.com/paper/2102.10809/) (2021)
- [Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare](https://mlsanity.com/paper/2202.01906/) (2022)
- [Fair Risk Control: A Generalized Framework for Calibrating Multi-group Fairness Risks](https://mlsanity.com/paper/2405.02225/) (2024)
- [Individually Fair Gradient Boosting](https://mlsanity.com/paper/2103.16785/) (2021)
- [A comparison of approaches to improve worst-case predictive model performance over patient subpopulations](https://mlsanity.com/paper/2108.12250/) (2021)
- [Bottlenecks CLUB: Unifying Information-Theoretic Trade-offs Among Complexity, Leakage, and Utility](https://mlsanity.com/paper/2207.04895/) (2022)
- [Fair admission risk prediction with proportional multicalibration](https://mlsanity.com/paper/2209.14613/) (2022)
- [Measuring Model Fairness under Noisy Covariates: A Theoretical Perspective](https://mlsanity.com/paper/2105.09985/) (2021)
- [Optimizing fairness tradeoffs in machine learning with multiobjective meta-models](https://mlsanity.com/paper/2304.12190/) (2023)
- [Precarity: Modeling the Long Term Effects of Compounded Decisions on Individual Instability](https://mlsanity.com/paper/2104.12037/) (2021)
- [Intersectional Unfairness Discovery](https://mlsanity.com/paper/2405.20790/) (2024)
- [Evaluating Model Performance Under Worst-case Subpopulations](https://mlsanity.com/paper/2407.01316/) (2024)
- [Learning With Multi-Group Guarantees For Clusterable Subpopulations](https://mlsanity.com/paper/2410.14588/) (2024)
- [Designing Equitable Algorithms](https://mlsanity.com/paper/2302.09157/) (2023)
- [Agnostic Multi-Robust Learning Using ERM](https://mlsanity.com/paper/2303.08944/) (2023)
- [Conformal Prediction Adaptive to Unknown Subpopulation Shifts](https://mlsanity.com/paper/2506.05583/) (2025)
- [Constrained Online Decision-Making: A Unified Framework](https://mlsanity.com/paper/2505.07101/) (2025)
- [Doubly-Regressing Approach for Subgroup Fairness](https://mlsanity.com/paper/2510.21091/) (2025)
- [On the Role of Calibration in Benchmarking Algorithmic Fairness for Skin Cancer Detection](https://mlsanity.com/paper/2511.07700/) (2025)
- [Revisiting Technical Bias Mitigation Strategies](https://mlsanity.com/paper/2410.17433/) (2024)

## Abstract (arXiv)

As algorithms increasingly inform and influence decisions made about individuals, it becomes increasingly important to address concerns that these algorithms might be discriminatory. The output of an algorithm can be discriminatory for many reasons, most notably: (1) the data used to train the algorithm might be biased (in various ways) to favor certain populations over others; (2) the analysis of this training data might inadvertently or maliciously introduce biases that are not borne out in the data. This work focuses on the latter concern. We develop and study multicalbration -- a new measure of algorithmic fairness that aims to mitigate concerns about discrimination that is introduced in the process of learning a predictor from data. Multicalibration guarantees accurate (calibrated) predictions for every subpopulation that can be identified within a specified class of computations. We think of the class as being quite rich; in particular, it can contain many overlapping subgroups of a protected group. We show that in many settings this strong notion of protection from discrimination is both attainable and aligned with the goal of obtaining accurate predictions. Along the way, we present new algorithms for learning a multicalibrated predictor, study the computational complexity of this task, and draw new connections to computational learning models such as agnostic learning.