mlsanityData to 14 Oct 2026

Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defer

arXiv 2506.20650 ↗First posted 25 Jun 2025cs.LG stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

The problem of learning to defer with multiple experts consists of optimally assigning input instances to experts, balancing the trade-off between their accuracy and computational cost. This is a critical challenge in natural language generation, but also in other fields such as image processing, and medical diagnostics. Recent studies have proposed surrogate loss functions to optimize deferral, but challenges remain in ensuring their consistency properties. This paper introduces novel surrogate loss functions and efficient algorithms with strong theoretical learning guarantees. We address open questions regarding realizable H-consistency, H-consistency bounds, and Bayes-consistency for both single-stage (jointly learning predictor and deferral function) and two-stage (learning only the deferral function with a fixed expert) learning scenarios. For single-stage deferral, we introduce a family of new realizable H-consistent surrogate losses and further prove H-consistency for a selected member. For two-stage deferral, we derive new surrogate losses that achieve realizable H-consistency, H-consistency bounds, and Bayes-consistency for the two-expert scenario and, under natural assumptions, multiple-expert scenario. Additionally, we provide enhanced theoretical guarantees under low-noise assumptions for both scenarios. Finally, we report the results of experiments using our proposed surrogate losses, comparing their performance against existing baselines.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 62 of its 116 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

62 of 116 references.

References · 62

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  5. 2017Selective Classification for Deep Neural Networkscited by 852
  6. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
  7. 2018Learning Adversarially Fair and Transferable Representationscited by 353
  8. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  9. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  10. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 210
  11. 2020Learning to Complement Humanscited by 124
  12. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
38 more
  1. 2021Differentiable Learning Under Triagecited by 76
  2. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 72
  3. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 70
  4. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 69
  5. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 60
  6. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
  7. 2020Classification with Rejection Based on Cost-sensitive Classificationcited by 55
  8. 2021Calibration and Consistency of Adversarial Surrogate Lossescited by 47
  9. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  10. 2020Classification Under Human Assistancecited by 42
  11. 2019On the Calibration of Multiclass Classification with Rejectioncited by 41
  12. 2019Regression Under Human Assistancecited by 40
  13. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  14. 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
  15. 2020Selective Classification via One-Sided Predictioncited by 38
  16. 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
  17. 2021Human-AI Collaboration with Bandit Feedbackcited by 36
  18. 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
  19. 2021Selective Regression Under Fairness Criteriacited by 34
  20. 2023Learning to Reject with a Fixed Predictor: Application to Decontextualizationcited by 32
  21. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  22. 2020Regression with reject option and application to kNNcited by 30
  23. 2021Teaching Humans When To Defer to a Classifier via Exemplarscited by 29
  24. 2023Revisiting Discriminative vs. Generative Classifiers: Theory and Implicationscited by 29
  25. 2024Regression with Multi-Expert Deferralcited by 29
  26. 2023Ranking with Abstentioncited by 27
  27. 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 27
  28. 2021A Finer Calibration Analysis for Adversarial Robustnesscited by 26
  29. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  30. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  31. 2023Learning to Defer with Limited Expert Predictionscited by 25
  32. 2024A Universal Growth Rate for Learning with Smooth Surrogate Lossescited by 24
  33. 2024Cardinality-Aware Set Prediction and Top-k Classificationcited by 23
  34. 2024Multi-Label Learning with Stronger Consistency Guaranteescited by 22
  35. 2024H-Consistency Guarantees for Regressioncited by 21
  36. 2021Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possiblecited by 19
  37. 2024Enhanced H-Consistency Boundscited by 18
  38. 2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 17

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

  1. 2025Budgeted Multiple-Expert Deferralcited by 13
  2. 2025Improved Balanced Classification with Theoretically Grounded Loss Functionscited by 9
  3. 2025Beyond Tsybakov: Model Margin Noise and H-Consistency Boundscited by 7
  4. 2025Principled Algorithms for Optimizing Generalized Metrics in Binary Classificationcited by 6
  5. 2026Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Modelscited by 6
  6. 2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
  7. 2026A Theoretical Framework for Modular Learning of Robust Generative Modelscited by 4
  8. 2026When More Experts Hurt: Underfitting in Multi-Expert Learning to Defercited by 3
  9. 2026Optimized Deferral for Imbalanced Settingscited by 2
  10. 2026Linear-Core Surrogates: Smooth Loss Functions with Linear Rates for Classification and Structured Predictioncited by 2
  11. 2026Online Learning-to-Defer with Varying Expertscited by 1
  12. 2026Density-Ratio Losses for Post-Hoc Learning to Defercited by 1
4 more
  1. 2026Principled Algorithms for Optimizing Generalized Metrics in Multi-Label Learningcited by 0
  2. 2026Human-AI Teaming Through the Lens of Calibrationcited by 0
  3. 2026Candidate-Expanding Routing with Permutation-Stabilized Experts for Mixed-Format Medical VQAcited by 0
  4. 2026Should I stay or should I show? Learning to selectively disclose informationcited by 0