Data to 7 Oct 2026

Principled Approaches for Learning to Defer with Multiple Experts

arXiv 2310.14774 ↗First posted 23 Oct 2023cs.LG stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

We present a study of surrogate losses and algorithms for the general problem of learning to defer with multiple experts. We first introduce a new family of surrogate losses specifically tailored for the multiple-expert setting, where the prediction and deferral functions are learned simultaneously. We then prove that these surrogate losses benefit from strong H-consistency bounds. We illustrate the application of our analysis through several examples of practical surrogate losses, for which we give explicit guarantees. These loss functions readily lead to the design of new learning to defer algorithms based on their minimization. While the main focus of this work is a theoretical analysis, we also report the results of several experiments on SVHN and CIFAR-10 datasets.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 21 of its 65 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

21 of 65 references.

References · 21

2015Deep Residual Learning for Image Recognitioncited by 51,809
2014Adam: A Method for Stochastic Optimizationcited by 36,960
2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
2018Learning Adversarially Fair and Transferable Representationscited by 353
2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
2020Consistent Estimators for Learning to Defer to an Expertcited by 262
2020Learning to Complement Humanscited by 123
2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
2021Differentiable Learning Under Triagecited by 74
2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 71
2019Deep Gamblers: Learning to Abstain with Portfolio Theorycited by 69
2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59

9 more

2019Regression Under Human Assistancecited by 40
2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
2020Selective Classification via One-Sided Predictioncited by 37
2021Human-AI Collaboration with Bandit Feedbackcited by 36
2018Investigating Human + Machine Complementarity for Recidivism Predictionscited by 19
2021Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possiblecited by 19
2021Reinforcement Learning Under Algorithmic Triagecited by 15
2021Improving Learning-to-Defer Algorithms Through Fine-Tuningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 14; 2026: 11.

38 papers.

Sort:

By area:

2025Universal Model Routing for Efficient LLM Inferencecited by 80
2024Regression with Multi-Expert Deferralcited by 29
2024Learning to Defer to a Population: A Meta-Learning Approachcited by 26
2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
2024A Universal Growth Rate for Learning with Smooth Surrogate Lossescited by 24
2024Cardinality-Aware Set Prediction and Top-k Classificationcited by 23
2024Multi-Label Learning with Stronger Consistency Guaranteescited by 22
2024H-Consistency Guarantees for Regressioncited by 21
2024Enhanced H-Consistency Boundscited by 18
2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 16
2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15

26 more

2025Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Datacited by 15
2025Budgeted Multiple-Expert Deferralcited by 11
2025Improved Balanced Classification with Theoretically Grounded Loss Functionscited by 9
2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
2025Beyond Tsybakov: Model Margin Noise and H-Consistency Boundscited by 7
2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 6
2025Principled Algorithms for Optimizing Generalized Metrics in Binary Classificationcited by 6
2023Learning to Complement with Multiple Humanscited by 5
2024Top-k Classification and Cardinality-Aware Predictioncited by 5
2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
2025Identity-Free Deferral For Unseen Expertscited by 4
2026Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Modelscited by 4
2026A Theoretical Framework for Modular Learning of Robust Generative Modelscited by 4
2025Learning to Help in Multi-Class Settingscited by 2
2025Conformal Set-based Human-AI Complementarity with Multiple Expertscited by 2
2026When More Experts Hurt: Underfitting in Multi-Expert Learning to Defercited by 2
2026Optimized Deferral for Imbalanced Settingscited by 2
2026Linear-Core Surrogates: Smooth Loss Functions with Linear Rates for Classification and Structured Predictioncited by 2
2025From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascadescited by 1
2026Consistent Learning-to-Defer with Expert-Conditional Advicecited by 1
2026Density-Ratio Losses for Post-Hoc Learning to Defercited by 1
2025Is Uncertainty Quantification a Viable Alternative to Learned Deferral?cited by 0
2026MPD2-Router: Mask-aware Multi-expert Prior-regularized Dual-head Deferral Router in Glaucoma Screening and Diagnosiscited by 0
2026Online Learning-to-Defer with Varying Expertscited by 0
2026Principled Algorithms for Optimizing Generalized Metrics in Multi-Label Learningcited by 0
2026One Human, N Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidencecited by 0