mlsanityData to 7 Oct 2026

Learning to Complement Humans

arXiv 2005.00582 ↗First posted 1 May 2020cs.AI cs.LGcited by 123

Not on the map. Why?

Abstract (arXiv)

A rising vision for AI in the open world centers on the development of systems that can complement humans for perceptual, diagnostic, and reasoning tasks. To date, systems aimed at complementing the skills of people have employed models trained to be as accurate as possible in isolation. We demonstrate how an end-to-end learning strategy can be harnessed to optimize the combined performance of human-machine teams by considering the distinct abilities of people and machines. The goal is to focus machine learning on problem instances that are difficult for humans, while recognizing instances that are difficult for the machine and seeking human input on them. We demonstrate in two real-world domains (scientific discovery and medical diagnosis) that human-machine teams built via these methods outperform the individual performance of machines and people. We then analyze conditions under which this complementarity is strongest, and which training methods amplify it. Taken together, our work provides the first systematic investigation of how machine learning systems can be trained to complement human reasoning.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 24; 2023: 24; 2024: 21; 2025: 12; 2026: 24.

123 papers.

Sort:

By area:

  1. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  2. 2021Learning Optimal Conformal Classifierscited by 108
  3. 2022CARD: Classification and Regression Diffusion Modelscited by 102
  4. 2024When combinations of humans and AI are useful: A systematic review and meta-analysiscited by 100
  5. 2021Differentiable Learning Under Triagecited by 74
  6. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 71
  7. 2023Evaluating Language Models for Mathematics through Interactionscited by 69
  8. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 68
  9. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 68
  10. 2023Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Makingcited by 67
  11. 2022Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderationcited by 66
  12. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
38 more
  1. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  2. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  3. 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
  4. 2023Human Uncertainty in Concept-Based AI Systemscited by 39
  5. 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
  6. 2024Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Makingcited by 38
  7. 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
  8. 2024Complementarity in Human-AI Collaboration: Concept, Sources, and Evidencecited by 35
  9. 2022On the Utility of Prediction Sets in Human-AI Teamscited by 34
  10. 2021Role of Human-AI Interaction in Selective Predictioncited by 33
  11. 2022Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Makingcited by 33
  12. 2022Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imagingcited by 32
  13. 2023Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfactioncited by 32
  14. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  15. 2021How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Studycited by 29
  16. 2021Teaching Humans When To Defer to a Classifier via Exemplarscited by 29
  17. 2022Uncalibrated Models Can Improve Human-AI Collaborationcited by 26
  18. 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 26
  19. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  20. 2023Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Makingcited by 25
  21. 2023Learning to Defer with Limited Expert Predictionscited by 25
  22. 2024Towards Human-AI Complementarity with Prediction Setscited by 25
  23. 2023The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputscited by 23
  24. 2022A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementaritycited by 22
  25. 2021How Much Automation Does a Data Scientist Want?cited by 20
  26. 2021Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possiblecited by 19
  27. 2021Designing for human-AI complementarity in K-12 educationcited by 19
  28. 2022Toward Supporting Perceptual Complementarity in Human-AI Collaboration via Reflection on Unobservablescited by 19
  29. 2022Learning When to Advise Human Decision Makerscited by 18
  30. 2021Leveraging Expert Consistency to Improve Algorithmic Decision Supportcited by 17
  31. 2022Human-AI Collaboration in Decision-Making: Beyond Learning to Defercited by 16
  32. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  33. 2021Reinforcement Learning Under Algorithmic Triagecited by 15
  34. 2023Learning Personalized Decision Support Policiescited by 13
  35. 2024Human Expertise in Algorithmic Predictioncited by 13
  36. 2025Budgeted Multiple-Expert Deferralcited by 11
  37. 2021Learning-to-defer for sequential medical decision-making under uncertaintycited by 10
  38. 2021Improving Learning-to-Defer Algorithms Through Fine-Tuningcited by 10

50 most cited of 123 All 123 (JSON)