mlsanityData to 7 Oct 2026

Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles

arXiv 2210.16955 ↗First posted 30 Oct 2022stat.ML cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

We study the statistical properties of learning to defer (L2D) to multiple experts. In particular, we address the open problems of deriving a consistent surrogate loss, confidence calibration, and principled ensembling of experts. Firstly, we derive two consistent surrogates -- one based on a softmax parameterization, the other on a one-vs-all (OvA) parameterization -- that are analogous to the single expert losses proposed by Mozannar and Sontag (2020) and Verma and Nalisnick (2022), respectively. We then study the frameworks' ability to estimate P( m_j = y | x ), the probability that the jth expert will correctly predict the label for x. Theory shows the softmax-based loss causes mis-calibration to propagate between the estimates while the OvA-based loss does not (though in practice, we find there are trade offs). Lastly, we propose a conformal inference technique that chooses a subset of experts to query when the system defers. We perform empirical validation on tasks for galaxy, skin lesion, and hate speech classification.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 29 of its 56 references that are in the map. First posted on arXiv 30 Oct 2022.

Builds on

29 of 56 references.

References · 29

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2016Wide Residual Networkscited by 3,057
  3. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  4. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  5. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  6. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  7. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  8. 2016FastText.zip: Compressing text classification modelscited by 331
  9. 2022Conformal Risk Controlcited by 290
  10. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  11. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  12. 2018To Trust Or Not To Trust A Classifiercited by 205
17 more
  1. 2019Evaluating model calibration in classificationcited by 165
  2. 2020Learning to Complement Humanscited by 123
  3. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
  4. 2021Differentiable Learning Under Triagecited by 74
  5. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 71
  6. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 68
  7. 2020Classification with Rejection Based on Cost-sensitive Classificationcited by 55
  8. 2021Hybrid Intelligencecited by 55
  9. 2021Top-label calibration and multiclass-to-binary reductionscited by 46
  10. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  11. 2019On the Calibration of Multiclass Classification with Rejectioncited by 41
  12. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  13. 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
  14. 2022On the Utility of Prediction Sets in Human-AI Teamscited by 34
  15. 2021Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possiblecited by 19
  16. 2021Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosiscited by 15
  17. 2022Counterfactual Inference of Second Opinionscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 14; 2025: 19; 2026: 19.

59 papers.

Sort:

By area:

  1. 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
  2. 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
  3. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  4. 2024Regression with Multi-Expert Deferralcited by 29
  5. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  6. 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 26
  7. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  8. 2024A Universal Growth Rate for Learning with Smooth Surrogate Lossescited by 24
  9. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  10. 2025Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answeringcited by 17
  11. 2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 16
  12. 2025Adversarial Robustness in Two-Stage Learning-to-Defer: Algorithms and Guaranteescited by 16
38 more
  1. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  2. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  3. 2024A Causal Framework for Evaluating Deferring Systemscited by 13
  4. 2024Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?cited by 13
  5. 2024Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraintscited by 12
  6. 2025Budgeted Multiple-Expert Deferralcited by 11
  7. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  8. 2024Learning to Complement and to Defer to Multiple Userscited by 7
  9. 2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 6
  10. 2023Learning to Complement with Multiple Humanscited by 5
  11. 2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
  12. 2024LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learningcited by 4
  13. 2025Learning to Partially Defer for Sequencescited by 4
  14. 2025Identity-Free Deferral For Unseen Expertscited by 4
  15. 2025To Ask or Not to Ask: Learning to Require Human Feedbackcited by 4
  16. 2026Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Modelscited by 4
  17. 2023FiFAR: A Fraud Detection Dataset for Learning to Defercited by 3
  18. 2026Fatigue-Aware Learning to Defer via Constrained Optimisationcited by 3
  19. 2023When No-Rejection Learning is Consistent for Regression with Rejectioncited by 2
  20. 2024Learning To Help: Training Models to Assist Legacy Devicescited by 2
  21. 2025Learning to Help in Multi-Class Settingscited by 2
  22. 2025Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregationcited by 2
  23. 2025No Need for Learning to Defer? A Training Free Deferral Framework to Multiple Experts through Conformal Predictioncited by 2
  24. 2026When More Experts Hurt: Underfitting in Multi-Expert Learning to Defercited by 2
  25. 2026Optimized Deferral for Imbalanced Settingscited by 2
  26. 2024Predicting Probabilities of Error to Combine Quantization and Early Exiting: QuEEcited by 1
  27. 2025Bayesian Inference for Correlated Human Experts and Classifierscited by 1
  28. 2025From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascadescited by 1
  29. 2026Consistent Learning-to-Defer with Expert-Conditional Advicecited by 1
  30. 2026DeferredSeg:A Multi-Expert Deferral Framework for Medical Image Segmentationcited by 1
  31. 2026Density-Ratio Losses for Post-Hoc Learning to Defercited by 1
  32. 2026Strategic Decision Support for AI Agentscited by 1
  33. 2026Calibration Is Not Control: Why LLM-Agent Oversight Needs Interventioncited by 1
  34. 2026RouteGuard: Certifying Routing Gain in LLM Multi-Agent Systems When Complementarity Is Not Enoughcited by 1
  35. 2025DeCoDe: Defer-and-Complement Decision-Making via Decoupled Concept Bottleneck Modelscited by 0
  36. 2025Rethinking Calibration for Early-Exit Neural Networkscited by 0
  37. 2025Know What You Don't Know: Selective Prediction for Early Exit DNNscited by 0
  38. 2025Learning To Defer To A Population With Limited Demonstrationscited by 0

50 most cited of 59 All 59 (JSON)