mlsanityData to 7 Oct 2026

Consistent Estimators for Learning to Defer to an Expert

arXiv 2006.01862 ↗First posted 2 Jun 2020cs.LG cs.HC stat.MLcited by 262

Not on the map. Why?

Cited by 20 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms. In this paper we explore how to learn predictors that can either predict or choose to defer the decision to a downstream expert. Given only samples of the expert's decisions, we give a procedure based on learning a classifier and a rejector and analyze it theoretically. Our approach is based on a novel reduction to cost sensitive learning where we give a consistent surrogate loss for cost sensitive learning that generalizes the cross entropy loss. We show the effectiveness of our approach on a variety of experimental tasks.

Key facts. Cited by 262 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 21; 2023: 38; 2024: 41; 2025: 37; 2026: 105.

262 papers.

Sort:

By area:

  1. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  2. 2021Machine Learning with a Reject Option: A surveycited by 107
  3. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  4. 2021Differentiable Learning Under Triagecited by 74
  5. 2024Building Machines that Learn and Think with Peoplecited by 73
  6. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  7. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 71
  8. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 68
  9. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 68
  10. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  11. 2021Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularizationcited by 62
  12. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 61
38 more
  1. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
  2. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
  3. 2022Improving Expert Predictions with Conformal Predictioncited by 54
  4. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  5. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 50
  6. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  7. 2022Predictive Multiplicity in Probabilistic Classificationcited by 44
  8. 2022The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanationscited by 41
  9. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 41
  10. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  11. 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
  12. 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
  13. 2023Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimationcited by 38
  14. 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 38
  15. 2026Building Production-Ready Probes For Geminicited by 38
  16. 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
  17. 2024Complementarity in Human-AI Collaboration: Concept, Sources, and Evidencecited by 35
  18. 2022On the Utility of Prediction Sets in Human-AI Teamscited by 34
  19. 2021Role of Human-AI Interaction in Selective Predictioncited by 33
  20. 2023Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfactioncited by 32
  21. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  22. 2021No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasetscited by 31
  23. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 31
  24. 2022Calibrated Selective Classificationcited by 30
  25. 2021Teaching Humans When To Defer to a Classifier via Exemplarscited by 29
  26. 2024Regression with Multi-Expert Deferralcited by 29
  27. 2022Uncalibrated Models Can Improve Human-AI Collaborationcited by 26
  28. 2023Designing Decision Support Systems Using Counterfactual Prediction Setscited by 26
  29. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  30. 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 26
  31. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  32. 2023Learning to Defer with Limited Expert Predictionscited by 25
  33. 2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
  34. 2024Towards Human-AI Complementarity with Prediction Setscited by 25
  35. 2024Evaluating Human-AI Collaboration: A Review and Methodological Frameworkcited by 25
  36. 2024A Universal Growth Rate for Learning with Smooth Surrogate Lossescited by 24
  37. 2023The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputscited by 23
  38. 2022A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementaritycited by 22

50 most cited of 262 All 262 (JSON)