mlsanityData to 7 Oct 2026

The Algorithmic Automation Problem: Prediction, Triage, and Human Effort

arXiv 1903.12220 ↗First posted 28 Mar 2019cs.CV cs.AI cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

In a wide array of areas, algorithms are matching and surpassing the performance of human experts, leading to consideration of the roles of human judgment and algorithmic prediction in these domains. The discussion around these developments, however, has implicitly equated the specific task of prediction with the general task of automation. We argue here that automation is broader than just a comparison of human versus algorithmic performance on a task; it also involves the decision of which instances of the task to give to the algorithm in the first place. We develop a general framework that poses this latter decision as an optimization problem, and we show how basic heuristics for this optimization problem can lead to performance gains even on heavily-studied applications of AI in medicine. Our framework also serves to highlight how effective automation depends crucially on estimating both algorithmic and human error on an instance-by-instance basis, and our results show how improvements in these error estimation problems can yield significant gains for automation as well.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 17; 2023: 18; 2024: 25; 2025: 14; 2026: 15.

101 papers.

Sort:

By area:

  1. 2022CARD: Classification and Regression Diffusion Modelscited by 102
  2. 2021Differentiable Learning Under Triagecited by 74
  3. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 71
  4. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 68
  5. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 68
  6. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
  7. 2022Improving Expert Predictions with Conformal Predictioncited by 54
  8. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  9. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  10. 2022Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairnesscited by 45
  11. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  12. 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
38 more
  1. 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
  2. 2022Visual correspondence-based explanations improve AI robustness and human-AI team accuracycited by 37
  3. 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
  4. 2022Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imagingcited by 32
  5. 2023Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfactioncited by 32
  6. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  7. 2021Teaching Humans When To Defer to a Classifier via Exemplarscited by 29
  8. 2023Designing Decision Support Systems Using Counterfactual Prediction Setscited by 26
  9. 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 26
  10. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  11. 2023Learning to Defer with Limited Expert Predictionscited by 25
  12. 2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
  13. 2024Towards Human-AI Complementarity with Prediction Setscited by 25
  14. 2022A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementaritycited by 22
  15. 2023Effective Human-AI Teams via Learned Natural Language Rules and Onboardingcited by 20
  16. 2021Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possiblecited by 19
  17. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  18. 2021Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosiscited by 15
  19. 2021Reinforcement Learning Under Algorithmic Triagecited by 15
  20. 2024Human Expertise in Algorithmic Predictioncited by 13
  21. 2024A Causal Framework for Evaluating Deferring Systemscited by 13
  22. 2025Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Modelscited by 13
  23. 2023Auditing for Human Expertisecited by 12
  24. 2024A2C: A Modular Multi-stage Collaborative Decision Framework for Human-AI Teamscited by 12
  25. 2024Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraintscited by 12
  26. 2022Algorithmic Assistance with Recommendation-Dependent Preferencescited by 11
  27. 2025Budgeted Multiple-Expert Deferralcited by 11
  28. 2021Learning-to-defer for sequential medical decision-making under uncertaintycited by 10
  29. 2021On the Fairness of Machine-Assisted Human Decisionscited by 10
  30. 2021Improving Learning-to-Defer Algorithms Through Fine-Tuningcited by 10
  31. 2024Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Setscited by 9
  32. 2023Learning Complementary Policies for Human-AI Teamscited by 7
  33. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  34. 2024Learning to Complement and to Defer to Multiple Userscited by 7
  35. 2025Towards Human-AI Complementarity in Matching Taskscited by 7
  36. 2024Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Modelcited by 6
  37. 2025Explaining and Improving Information Complementarities in Multi-Agent Decision-makingcited by 6
  38. 2023Learning to Complement with Multiple Humanscited by 5

50 most cited of 101 All 101 (JSON)