# Uncalibrated Models Can Improve Human-AI Collaboration

arXiv: https://arxiv.org/abs/2202.05983 · HTML: https://mlsanity.com/paper/2202.05983/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 45 references that are in the map.
- First posted on arXiv 12 Feb 2022.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [A Large Self-Annotated Corpus for Sarcasm](https://mlsanity.com/paper/1704.05579/) (2017)
- [CheXplain: Enabling Physicians to Explore and UnderstandData-Driven, AI-Enabled Medical Imaging Analysis](https://mlsanity.com/paper/2001.05149/) (2020)
- [Do Humans Trust Advice More if it Comes from AI? An Analysis of Human-AI Interactions](https://mlsanity.com/paper/2107.07015/) (2021)
- [TrueImage: A Machine Learning Algorithm to Improve the Quality of Telehealth Photos](https://mlsanity.com/paper/2010.02086/) (2020)

## Built on by (most cited first, 26 of 26)

- [Large Language Models Must Be Taught to Know What They Don't Know](https://mlsanity.com/paper/2406.08391/) (2024)
- [Forecasting Future World Events with Neural Networks](https://mlsanity.com/paper/2206.15474/) (2022)
- [Introducing v0.5 of the AI Safety Benchmark from MLCommons](https://mlsanity.com/paper/2404.12241/) (2024)
- [Eliciting and Learning with Soft Labels from Every Annotator](https://mlsanity.com/paper/2207.00810/) (2022)
- [Human-Aligned Calibration for AI-Assisted Decision Making](https://mlsanity.com/paper/2306.00074/) (2023)
- [Designing Decision Support Systems Using Counterfactual Prediction Sets](https://mlsanity.com/paper/2306.03928/) (2023)
- [Towards Human-AI Complementarity with Prediction Sets](https://mlsanity.com/paper/2405.17544/) (2024)
- [Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors](https://mlsanity.com/paper/2310.16924/) (2023)
- [From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered](https://mlsanity.com/paper/2506.07461/) (2025)
- [Effective Human-AI Teams via Learned Natural Language Rules and Onboarding](https://mlsanity.com/paper/2311.01007/) (2023)
- [Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?](https://mlsanity.com/paper/2402.16102/) (2024)
- [Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary](https://mlsanity.com/paper/2411.10461/) (2024)
- [Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets](https://mlsanity.com/paper/2406.06671/) (2024)
- [The Impact and Feasibility of Self-Confidence Shaping for AI-Assisted Decision-Making](https://mlsanity.com/paper/2502.14311/) (2025)
- [Beyond Accuracy: How AI Metacognitive Sensitivity improves AI-assisted Decision Making](https://mlsanity.com/paper/2507.22365/) (2025)
- [L2CU: Learning to Complement Unseen Users](https://mlsanity.com/paper/2601.06119/) (2026)
- [Optimising Human-AI Collaboration by Learning Convincing Explanations](https://mlsanity.com/paper/2311.07426/) (2023)
- [A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessment](https://mlsanity.com/paper/2408.01301/) (2024)
- [Human-AI Collaborative Game Testing with Vision Language Models](https://mlsanity.com/paper/2501.11782/) (2025)
- [Human-AI Collaboration: Trade-offs Between Performance and Preferences](https://mlsanity.com/paper/2503.00248/) (2025)
- [Does Calibration Affect Human Actions?](https://mlsanity.com/paper/2508.18317/) (2025)
- [Human-LLM Collaborative Feature Engineering for Tabular Data](https://mlsanity.com/paper/2601.21060/) (2026)
- [Learning to Decide with AI Assistance under Human-Alignment](https://mlsanity.com/paper/2605.12646/) (2026)
- [Calibrating Ensembles for Scalable Uncertainty Quantification in Deep Learning-based Medical Segmentation](https://mlsanity.com/paper/2209.09563/) (2022)
- [Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations](https://mlsanity.com/paper/2509.25844/) (2025)
- [Human-AI Teaming Through the Lens of Calibration](https://mlsanity.com/paper/2606.10906/) (2026)

## Abstract (arXiv)

In many practical applications of AI, an AI model is used as a decision aid for human users. The AI provides advice that a human (sometimes) incorporates into their decision-making process. The AI advice is often presented with some measure of "confidence" that the human can use to calibrate how much they depend on or trust the advice. In this paper, we present an initial exploration that suggests showing AI models as more confident than they actually are, even when the original AI is well-calibrated, can improve human-AI performance (measured as the accuracy and confidence of the human's final prediction after seeing the AI advice). We first train a model to predict human incorporation of AI advice using data from thousands of human-AI interactions. This enables us to explicitly estimate how to transform the AI's prediction confidence, making the AI uncalibrated, in order to improve the final human prediction. We empirically validate our results across four different tasks--dealing with images, text and tabular data--involving hundreds of human participants. We further support our findings with simulation analysis. Our findings suggest the importance of jointly optimizing the human-AI system as opposed to the standard paradigm of optimizing the AI model alone.