# Human-Aligned Calibration for AI-Assisted Decision Making

arXiv: https://arxiv.org/abs/2306.00074 · HTML: https://mlsanity.com/paper/2306.00074/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons](https://mlsanity.com/paper/2301.11270/) (2023)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration](https://mlsanity.com/paper/2107.05719/) (2021)
- [A Large Self-Annotated Corpus for Sarcasm](https://mlsanity.com/paper/1704.05579/) (2017)
- [Improving Expert Predictions with Conformal Prediction](https://mlsanity.com/paper/2201.12006/) (2022)
- [How model accuracy and explanation fidelity influence user trust](https://mlsanity.com/paper/1907.12652/) (2019)
- [Distribution-free calibration guarantees for histogram binning without sample splitting](https://mlsanity.com/paper/2105.04656/) (2021)
- [Do Humans Trust Advice More if it Comes from AI? An Analysis of Human-AI Interactions](https://mlsanity.com/paper/2107.07015/) (2021)
- [The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems](https://mlsanity.com/paper/2008.09100/) (2020)
- [Uncalibrated Models Can Improve Human-AI Collaboration](https://mlsanity.com/paper/2202.05983/) (2022)
- [Machine Explanations and Human Understanding](https://mlsanity.com/paper/2202.04092/) (2022)
- [Improving Screening Processes via Calibrated Subset Selection](https://mlsanity.com/paper/2202.01147/) (2022)
- [Enhancing the Accuracy and Fairness of Human Decision Making](https://mlsanity.com/paper/1805.10318/) (2018)
- [Useful Confidence Measures: Beyond the Max Score](https://mlsanity.com/paper/2210.14070/) (2022)
- [Decision-Making under Miscalibration](https://mlsanity.com/paper/2203.09852/) (2022)

## Built on by (most cited first, 28 of 28)

- [Towards Human-AI Complementarity with Prediction Sets](https://mlsanity.com/paper/2405.17544/) (2024)
- [From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered](https://mlsanity.com/paper/2506.07461/) (2025)
- [When is Multicalibration Post-Processing Necessary?](https://mlsanity.com/paper/2406.06487/) (2024)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Human Expertise in Algorithmic Prediction](https://mlsanity.com/paper/2402.00793/) (2024)
- [Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary](https://mlsanity.com/paper/2411.10461/) (2024)
- [Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets](https://mlsanity.com/paper/2406.06671/) (2024)
- [Conformal Prediction and Human Decision Making](https://mlsanity.com/paper/2503.11709/) (2025)
- [Effective Generative AI: The Human-Algorithm Centaur](https://mlsanity.com/paper/2406.10942/) (2024)
- [Towards Human-AI Complementarity in Matching Tasks](https://mlsanity.com/paper/2508.13285/) (2025)
- [Explaining and Improving Information Complementarities in Multi-Agent Decision-making](https://mlsanity.com/paper/2502.06152/) (2025)
- [Impact of Decentralized Learning on Player Utilities in Stackelberg Games](https://mlsanity.com/paper/2403.00188/) (2024)
- [Human-Alignment Influences the Utility of AI-assisted Decision Making](https://mlsanity.com/paper/2501.14035/) (2025)
- [When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-making](https://mlsanity.com/paper/2308.11721/) (2023)
- [Integrating Expert Judgment and Algorithmic Decision Making: An Indistinguishability Framework](https://mlsanity.com/paper/2410.08783/) (2024)
- [Narrowing Action Choices with AI Improves Human Sequential Decisions](https://mlsanity.com/paper/2510.16097/) (2025)
- [Successfully Guiding Humans with Imperfect Instructions by Highlighting Potential Errors and Suggesting Corrections](https://mlsanity.com/paper/2402.16973/) (2024)
- [h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective](https://mlsanity.com/paper/2506.17968/) (2025)
- [Beyond Accuracy: How AI Metacognitive Sensitivity improves AI-assisted Decision Making](https://mlsanity.com/paper/2507.22365/) (2025)
- [ComplLLM: Fine-tuning LLMs to Discover Complementary Signals for Decision-making](https://mlsanity.com/paper/2602.19458/) (2026)
- [Underspecified Human Decision Experiments Considered Harmful](https://mlsanity.com/paper/2401.15106/) (2024)
- [A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessment](https://mlsanity.com/paper/2408.01301/) (2024)
- [Bayesian Inference for Correlated Human Experts and Classifiers](https://mlsanity.com/paper/2506.05636/) (2025)
- [Learning to Decide with AI Assistance under Human-Alignment](https://mlsanity.com/paper/2605.12646/) (2026)
- [Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision](https://mlsanity.com/paper/2604.06723/) (2026)
- [LLM Nepotism in Organizational Governance](https://mlsanity.com/paper/2604.09620/) (2026)
- [Human-AI Teaming Through the Lens of Calibration](https://mlsanity.com/paper/2606.10906/) (2026)
- [Human-Centric Reflective Architecture for Human-AI Collaborative Decision-Making](https://mlsanity.com/paper/2607.03025/) (2026)

## Abstract (arXiv)

Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of empirical evidence suggest that decision makers have difficulties at developing a good sense on when to trust a prediction using these confidence values. In this paper, our goal is first to understand why and then investigate how to construct more useful confidence values. We first argue that, for a broad class of utility functions, there exist data distributions for which a rational decision maker is, in general, unlikely to discover the optimal decision policy using the above confidence values -- an optimal decision maker would need to sometimes place more (less) trust on predictions with lower (higher) confidence values. However, we then show that, if the confidence values satisfy a natural alignment property with respect to the decision maker's confidence on her own predictions, there always exists an optimal decision policy under which the level of trust the decision maker would need to place on predictions is monotone on the confidence values, facilitating its discoverability. Further, we show that multicalibration with respect to the decision maker's confidence on her own predictions is a sufficient condition for alignment. Experiments on four different AI-assisted decision making tasks where a classifier provides decision support to real human experts validate our theoretical results and suggest that alignment may lead to better decisions.