mlsanityData to 7 Oct 2026

Who Should Predict? Exact Algorithms For Learning to Defer to Humans

arXiv 2301.06197 ↗First posted 15 Jan 2023cs.LG cs.HCcited by 68

Not on the map. Why?

Abstract (arXiv)

Automated AI classifiers should be able to defer the prediction to a human decision maker to ensure more accurate predictions. In this work, we jointly train a classifier with a rejector, which decides on each data point whether the classifier or the human should predict. We show that prior approaches can fail to find a human-AI system with low misclassification error even when there exists a linear classifier and rejector that have zero error (the realizable setting). We prove that obtaining a linear pair with low error is NP-hard even when the problem is realizable. To complement this negative result, we give a mixed-integer-linear-programming (MILP) formulation that can optimally solve the problem in the linear setting. However, the MILP only scales to moderately-sized problems. Therefore, we provide a novel surrogate loss function that is realizable-consistent and performs well empirically. We test our approaches on a comprehensive set of datasets and compare to a wide range of baselines.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 27 of its 56 references that are in the map. First posted on arXiv 15 Jan 2023.

Builds on

27 of 56 references.

References · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2016Wide Residual Networkscited by 3,057
  5. 2021Convex optimizationcited by 2,479
  6. 2017Selective Classification for Deep Neural Networkscited by 799
  7. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  8. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  9. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  10. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  11. 2020Learning to Complement Humanscited by 123
  12. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
15 more
  1. 2015Supersparse Linear Integer Models for Optimized Medical Scoring Systemscited by 76
  2. 2021Differentiable Learning Under Triagecited by 74
  3. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 71
  4. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 68
  5. 2021Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Makingcited by 58
  6. 2020Classification with Rejection Based on Cost-sensitive Classificationcited by 55
  7. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  8. 2019Regression Under Human Assistancecited by 40
  9. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  10. 2020Selective Classification via One-Sided Predictioncited by 37
  11. 2021Teaching Humans When To Defer to a Classifier via Exemplarscited by 29
  12. 2021Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possiblecited by 19
  13. 2021Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosiscited by 15
  14. 2021Improving Learning-to-Defer Algorithms Through Fine-Tuningcited by 10
  15. 2020Budget Learning via Bracketingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 21; 2025: 20; 2026: 17.

68 papers.

Sort:

By area:

  1. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 61
  2. 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
  3. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  4. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  5. 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 26
  6. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  7. 2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
  8. 2024Towards Human-AI Complementarity with Prediction Setscited by 25
  9. 2024A Universal Growth Rate for Learning with Smooth Surrogate Lossescited by 24
  10. 2023Effective Human-AI Teams via Learned Natural Language Rules and Onboardingcited by 20
  11. 2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 16
  12. 2025Adversarial Robustness in Two-Stage Learning-to-Defer: Algorithms and Guaranteescited by 16
38 more
  1. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  2. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  3. 2024A Causal Framework for Evaluating Deferring Systemscited by 13
  4. 2024Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraintscited by 12
  5. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 11
  6. 2025Strategic Tradeoffs Between Humans and AI in Multi-Agent Bargainingcited by 11
  7. 2025Budgeted Multiple-Expert Deferralcited by 11
  8. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  9. 2024Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Setscited by 9
  10. 2023Fair Classifiers that Abstain without Harmcited by 8
  11. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  12. 2024Learning to Complement and to Defer to Multiple Userscited by 7
  13. 2025Towards Human-AI Complementarity in Matching Taskscited by 7
  14. 2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 6
  15. 2023Learning to Complement with Multiple Humanscited by 5
  16. 2024A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problemscited by 5
  17. 2024Online Decision Deferral under Budget Constraintscited by 5
  18. 2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
  19. 2026When to Trust the Cheap Check: Weak and Strong Verification for Reasoningcited by 5
  20. 2024LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learningcited by 4
  21. 2025Cost-Aware Routing for Efficient Text-To-Image Generationcited by 4
  22. 2025To Ask or Not to Ask: Learning to Require Human Feedbackcited by 4
  23. 2026Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Modelscited by 4
  24. 2023FiFAR: A Fraud Detection Dataset for Learning to Defercited by 3
  25. 2024Partial-Label Learning with a Reject Optioncited by 3
  26. 2024Rejection via Learning Density Ratioscited by 3
  27. 2025Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrestcited by 3
  28. 2023When No-Rejection Learning is Consistent for Regression with Rejectioncited by 2
  29. 2024Toward A Causal Framework for Modeling Perceptioncited by 2
  30. 2024A Dynamic Model of Performative Human-ML Collaboration: Theory and Empirical Evidencecited by 2
  31. 2024Learning To Help: Training Models to Assist Legacy Devicescited by 2
  32. 2025Learning to Help in Multi-Class Settingscited by 2
  33. 2025Partial-Label Learning with Conformal Candidate Cleaningcited by 2
  34. 2025Selective Prediction Reduces the Negative Effects of Automation Bias Overall but Increases False Negativescited by 2
  35. 2025No Need for Learning to Defer? A Training Free Deferral Framework to Multiple Experts through Conformal Predictioncited by 2
  36. 2026L2CU: Learning to Complement Unseen Userscited by 2
  37. 2026Optimized Deferral for Imbalanced Settingscited by 2
  38. 2025A Comparison of Human and Machine Learning Errors in Face Recognitioncited by 1

50 most cited of 68 All 68 (JSON)