# Teaching Humans When To Defer to a Classifier via Exemplars

arXiv: https://arxiv.org/abs/2111.11297 · HTML: https://mlsanity.com/paper/2111.11297/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 84 references that are in the map.
- First posted on arXiv 22 Nov 2021.

## Builds on (most cited first, 30 of 40)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019)
- ["Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans](https://mlsanity.com/paper/2001.05871/) (2020)
- [How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation](https://mlsanity.com/paper/1802.00682/) (2018)
- [An Overview of Machine Teaching](https://mlsanity.com/paper/1801.05927/) (2018)
- [Direct Uncertainty Prediction for Medical Second Opinions](https://mlsanity.com/paper/1807.01771/) (2018)
- [Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents](https://mlsanity.com/paper/1911.00484/) (2019)
- [What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play](https://mlsanity.com/paper/1810.09648/) (2018)
- [Do Explanations make VQA Models more Predictable to a Human?](https://mlsanity.com/paper/1810.12366/) (2018)
- [Metric Learning for Individual Fairness](https://mlsanity.com/paper/1906.00250/) (2019)
- [CheXplain: Enabling Physicians to Explore and UnderstandData-Driven, AI-Enabled Medical Imaging Analysis](https://mlsanity.com/paper/2001.05149/) (2020)
- [Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction](https://mlsanity.com/paper/2007.12248/) (2020)
- [Near-Optimally Teaching the Crowd to Classify](https://mlsanity.com/paper/1402.2092/) (2014)
- [Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative](https://mlsanity.com/paper/2103.04725/) (2021)

## Built on by (most cited first, 29 of 29)

- [Who Should Predict? Exact Algorithms For Learning to Defer to Humans](https://mlsanity.com/paper/2301.06197/) (2023)
- [Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making](https://mlsanity.com/paper/2301.05809/) (2023)
- [Sample Efficient Learning of Predictors that Complement Humans](https://mlsanity.com/paper/2207.09584/) (2022)
- [Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Support](https://mlsanity.com/paper/2204.02310/) (2022)
- [Improving Human-AI Collaboration With Descriptions of AI Behavior](https://mlsanity.com/paper/2301.06937/) (2023)
- [In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defer](https://mlsanity.com/paper/2311.01106/) (2023)
- [Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defer](https://mlsanity.com/paper/2407.13732/) (2024)
- [Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making](https://mlsanity.com/paper/2302.06503/) (2023)
- [When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming](https://mlsanity.com/paper/2306.04930/) (2023)
- [Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions](https://mlsanity.com/paper/2302.07248/) (2023)
- [Effective Human-AI Teams via Learned Natural Language Rules and Onboarding](https://mlsanity.com/paper/2311.01007/) (2023)
- [Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools](https://mlsanity.com/paper/2303.00192/) (2023)
- [Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defer](https://mlsanity.com/paper/2506.20650/) (2025)
- [Learning Personalized Decision Support Policies](https://mlsanity.com/paper/2304.06701/) (2023)
- [Contrastive Explanations That Anticipate Human Misconceptions Can Improve Human Decision-Making Skills](https://mlsanity.com/paper/2410.04253/) (2024)
- [Budgeted Multiple-Expert Deferral](https://mlsanity.com/paper/2510.26706/) (2025)
- [Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary](https://mlsanity.com/paper/2411.10461/) (2024)
- [A No Free Lunch Theorem for Human-AI Collaboration](https://mlsanity.com/paper/2411.15230/) (2024)
- [To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems](https://mlsanity.com/paper/2409.14377/) (2024)
- [When Should We Orchestrate Multiple Agents?](https://mlsanity.com/paper/2503.13577/) (2025)
- [Training Towards Critical Use: Learning to Situate AI Predictions Relative to Human Knowledge](https://mlsanity.com/paper/2308.15700/) (2023)
- [Learning to Complement with Multiple Humans](https://mlsanity.com/paper/2311.13172/) (2023)
- [Online Decision Mediation](https://mlsanity.com/paper/2310.18601/) (2023)
- [Learning To Help: Training Models to Assist Legacy Devices](https://mlsanity.com/paper/2409.16253/) (2024)
- [Underspecified Human Decision Experiments Considered Harmful](https://mlsanity.com/paper/2401.15106/) (2024)
- [Trust Your Gut: Comparing Human and Machine Inference from Noisy Visualizations](https://mlsanity.com/paper/2407.16871/) (2024)
- [Rules or Weights? Comparing User Understanding of Explainable AI Techniques with the Cognitive XAI-Adaptive Model](https://mlsanity.com/paper/2602.19620/) (2026)
- [People-Centred Medical Image Analysis via Fairness-Aware Human-AI Cooperation](https://mlsanity.com/paper/2604.26991/) (2026)
- [CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations](https://mlsanity.com/paper/2604.27354/) (2026)

## Abstract (arXiv)

Expert decision makers are starting to rely on data-driven automated agents to assist them with various tasks. For this collaboration to perform properly, the human decision maker must have a mental model of when and when not to rely on the agent. In this work, we aim to ensure that human decision makers learn a valid mental model of the agent's strengths and weaknesses. To accomplish this goal, we propose an exemplar-based teaching strategy where humans solve the task with the help of the agent and try to formulate a set of guidelines of when and when not to defer. We present a novel parameterization of the human's mental model of the AI that applies a nearest neighbor rule in local regions surrounding the teaching examples. Using this model, we derive a near-optimal strategy for selecting a representative teaching set. We validate the benefits of our teaching strategy on a multi-hop question answering task using crowd workers and find that when workers draw the right lessons from the teaching stage, their task performance improves, we furthermore validate our method on a set of synthetic experiments.