mlsanityData to 14 Oct 2026

Sample Efficient Learning of Predictors that Complement Humans

arXiv 2207.09584 ↗First posted 19 Jul 2022cs.LG cs.AI cs.HCcited by 46

Not on the map. Why?

Abstract (arXiv)

One of the goals of learning algorithms is to complement and reduce the burden on human decision makers. The expert deferral setting wherein an algorithm can either predict on its own or defer the decision to a downstream expert helps accomplish this goal. A fundamental aspect of this setting is the need to learn complementary predictors that improve on the human's weaknesses rather than learning predictors optimized for average error. In this work, we provide the first theoretical analysis of the benefit of learning complementary predictors in expert deferral. To enable efficiently learning such predictors, we consider a family of consistent surrogate loss functions for expert deferral and analyze their theoretical properties. Finally, we design active learning schemes that require minimal amount of data of human expert predictions in order to learn accurate deferral systems.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 26 of its 46 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

26 of 46 references.

References · 26

  1. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  2. 2017Selective Classification for Deep Neural Networkscited by 852
  3. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  4. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  5. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  6. 2020Learning to Complement Humanscited by 124
  7. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
  8. 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86
  9. 2021Differentiable Learning Under Triagecited by 76
  10. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 72
  11. 2018An Overview of Machine Teachingcited by 63
  12. 2020Classification with Rejection Based on Cost-sensitive Classificationcited by 55
14 more
  1. 2017Active Learning for Cost-Sensitive Classificationcited by 41
  2. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  3. 2020Selective Classification via One-Sided Predictioncited by 38
  4. 2021Human-AI Collaboration with Bandit Feedbackcited by 36
  5. 2021Teaching Humans When To Defer to a Classifier via Exemplarscited by 29
  6. 2015Active Learning from Weak and Strong Labelerscited by 22
  7. 2018Investigating Human + Machine Complementarity for Recidivism Predictionscited by 19
  8. 2021Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possiblecited by 19
  9. 2021Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosiscited by 17
  10. 2021Reinforcement Learning Under Algorithmic Triagecited by 15
  11. 2019A General Theory for Structured Prediction with Smooth Convex Surrogatescited by 14
  12. 2021Improving Learning-to-Defer Algorithms Through Fine-Tuningcited by 11
  13. 2019Online Active Learning of Reject Option Classifierscited by 9
  14. 2020Budget Learning via Bracketingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 13; 2025: 10; 2026: 12.

46 papers.

Sort:

By area:

  1. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 70
  2. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 60
  3. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
  4. 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 27
  5. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  6. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  7. 2023Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Makingcited by 25
  8. 2023Learning to Defer with Limited Expert Predictionscited by 25
  9. 2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
  10. 2024Towards Human-AI Complementarity with Prediction Setscited by 25
  11. 2023Effective Human-AI Teams via Learned Natural Language Rules and Onboardingcited by 20
  12. 2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 17
34 more
  1. 2025Adversarial Robustness in Two-Stage Learning-to-Defer: Algorithms and Guaranteescited by 16
  2. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  3. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  4. 2023Learning Personalized Decision Support Policiescited by 14
  5. 2024A Causal Framework for Evaluating Deferring Systemscited by 13
  6. 2025Budgeted Multiple-Expert Deferralcited by 13
  7. 2024Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraintscited by 12
  8. 2024Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Setscited by 9
  9. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  10. 2024Learning to Complement and to Defer to Multiple Userscited by 7
  11. 2025When Should We Orchestrate Multiple Agents?cited by 7
  12. 2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 7
  13. 2025Towards Human-AI Complementarity in Matching Taskscited by 7
  14. 2025Human-AI Collaborative Uncertainty Quantificationcited by 7
  15. 2025To Ask or Not to Ask: Learning to Require Human Feedbackcited by 6
  16. 2026Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Modelscited by 6
  17. 2023Learning to Complement with Multiple Humanscited by 5
  18. 2024A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problemscited by 5
  19. 2024Online Decision Deferral under Budget Constraintscited by 5
  20. 2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
  21. 2026Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Humancited by 4
  22. 2026When More Experts Hurt: Underfitting in Multi-Expert Learning to Defercited by 3
  23. 2026Multi-Round Human-AI Collaboration with User-Specified Requirementscited by 3
  24. 2026Consistent Learning-to-Defer with Expert-Conditional Advicecited by 3
  25. 2023When No-Rejection Learning is Consistent for Regression with Rejectioncited by 2
  26. 2024A Dynamic Model of Performative Human-ML Collaboration: Theory and Empirical Evidencecited by 2
  27. 2025Conformal Set-based Human-AI Complementarity with Multiple Expertscited by 2
  28. 2026Optimized Deferral for Imbalanced Settingscited by 2
  29. 2026Calibrating conditional riskcited by 1
  30. 2026Online Learning-to-Defer with Varying Expertscited by 1
  31. 2026Strategic Decision Support for AI Agentscited by 1
  32. 2026The Confidence Gate Theorem: When Should Ranked Decision Systems Abstain?cited by 0
  33. 2026Overcoming the Incentive Collapse Paradoxcited by 0
  34. 2026Human-AI Teaming Through the Lens of Calibrationcited by 0