Consistent Learning-to-Defer with Expert-Conditional Advice
Not on the map. Why?
Abstract (arXiv)
We introduce learning-to-defer with advice, a new setting in which a system learns both which expert should handle an input and which information that expert should receive. The policy acts before observing advice and minimizes prediction loss plus expert and acquisition fees. Training may reveal all advice sources or only a subset on each record. We show that composing routing and advice losses can be inconsistent and that augmented training can incur avoidable estimation variance. We propose reference conditional classification, which jointly learns a fixed expert's error probability and each action's error conditional on that expert's success or failure. The reference outcome is observed on every labeled record, so records with missing advice still inform the estimates. We prove a consistency bound for bounded task losses, show that limited training preserves the full population objective, and establish variance reductions for the analyzed fits. On FEVER and Sensitive, full training has the lowest mean dev loss at every evaluated fee, while limited training improves on full training at most matched acquisition budgets.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 15 Mar 2026.
Builds on
35 of 57 references.
References · 35
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
- 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
- 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
- 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 72
- 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 70
- 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 60
- 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
23 more
- 2019On the Calibration of Multiclass Classification with Rejectioncited by 41
- 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
- 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
- 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
- 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
- 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
- 2024Regression with Multi-Expert Deferralcited by 29
- 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 27
- 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
- 2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 17
- 2024Trustworthy and Practical AI for Healthcare: A Guided Deferral System with Large Language Modelscited by 16
- 2025Adversarial Robustness in Two-Stage Learning-to-Defer: Algorithms and Guaranteescited by 16
- 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
- 2022Counterfactual Inference of Second Opinionscited by 14
- 2024A Causal Framework for Evaluating Deferring Systemscited by 13
- 2021Learning-to-defer for sequential medical decision-making under uncertaintycited by 12
- 2021Improving Learning-to-Defer Algorithms Through Fine-Tuningcited by 11
- 2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 7
- 2025To Ask or Not to Ask: Learning to Require Human Feedbackcited by 6
- 2026Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Modelscited by 6
- 2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
- 2025Identity-Free Deferral For Unseen Expertscited by 4
- 2021Finding the Optimal Dynamic Treatment Regime Using Smooth Fisher Consistent Surrogate Losscited by 2
Built on by
Citing papers per year: 2026: 3.
3 papers.