When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer
Not on the map. Why?
Abstract (arXiv)
Learning to Defer (L2D) enables a classifier to abstain from predictions and defer to an expert, and has recently been extended to multi-expert settings. In this work, we show that multi-expert L2D is fundamentally more challenging than the single-expert case. With multiple experts, the classifier's underfitting becomes inherent, which seriously degrades prediction performance, whereas in the single-expert setting it arises only under specific conditions. We theoretically reveal that this stems from an intrinsic expert identifiability issue: learning which expert to trust from a diverse pool, a problem absent in the single-expert case and renders existing underfitting remedies failed. To tackle this issue, we propose PiCCE (Pick the Confident and Correct Expert), a surrogate-based method that adaptively identifies a reliable expert based on empirical evidence. PiCCE effectively reduces multi-expert L2D to a single-expert-like learning problem, thereby resolving multi expert underfitting. We further prove its statistical consistency and ability to recover class probabilities and expert accuracies. Extensive experiments across diverse settings, including real-world expert scenarios, validate our theoretical results and demonstrate improved performance.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 19 Feb 2026.
Builds on
30 of 43 references.
References · 30
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2016Wide Residual Networkscited by 3,070
- 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
- 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
- 2025Universal Model Routing for Efficient LLM Inferencecited by 86
- 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 69
- 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 60
- 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
- 2020Classification with Rejection Based on Cost-sensitive Classificationcited by 55
- 2021Hard Sample Aware Noise Robust Learning for Histopathology Image Classificationcited by 48
18 more
- 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
- 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
- 2020Classification Under Human Assistancecited by 42
- 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
- 2024Faster Cascades via Speculative Decodingcited by 38
- 2023Ranking with Abstentioncited by 27
- 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 27
- 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
- 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
- 2023Learning to Defer with Limited Expert Predictionscited by 25
- 2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 17
- 2025Adversarial Robustness in Two-Stage Learning-to-Defer: Algorithms and Guaranteescited by 16
- 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
- 2023Plugin estimators for selective classification with out-of-distribution detectioncited by 14
- 2024A Causal Framework for Evaluating Deferring Systemscited by 13
- 2025Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Expertscited by 6
- 2023Learning to Complement with Multiple Humanscited by 5
- 2024A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problemscited by 5
Built on by
Citing papers per year: 2026: 3.
3 papers.