# Learning to Defer to a Population: A Meta-Learning Approach

arXiv: https://arxiv.org/abs/2403.02683 · HTML: https://mlsanity.com/paper/2403.02683/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 62 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Convolutional Conditional Neural Processes](https://mlsanity.com/paper/1910.13556/) (2019)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019)
- [Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes](https://mlsanity.com/paper/2007.01332/) (2020)
- [Differentiable Learning Under Triage](https://mlsanity.com/paper/2103.08902/) (2021)
- [Towards Unbiased and Accurate Deferral to Multiple Experts](https://mlsanity.com/paper/2102.13004/) (2021)
- [Who Should Predict? Exact Algorithms For Learning to Defer to Humans](https://mlsanity.com/paper/2301.06197/) (2023)
- [Calibrated Learning to Defer with One-vs-All Classifiers](https://mlsanity.com/paper/2202.03673/) (2022)
- [Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles](https://mlsanity.com/paper/2210.16955/) (2022)
- [Classification with Rejection Based on Cost-sensitive Classification](https://mlsanity.com/paper/2010.11748/) (2020)
- [Hybrid Intelligence](https://mlsanity.com/paper/2105.00691/) (2021)
- [TaskNorm: Rethinking Batch Normalization for Meta-Learning](https://mlsanity.com/paper/2003.03284/) (2020)
- [Sample Efficient Learning of Predictors that Complement Humans](https://mlsanity.com/paper/2207.09584/) (2022)
- [Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks](https://mlsanity.com/paper/1911.09737/) (2019)
- [On the Calibration of Multiclass Classification with Rejection](https://mlsanity.com/paper/1901.10655/) (2019)
- [Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts](https://mlsanity.com/paper/2206.07948/) (2022)
- [Principled Approaches for Learning to Defer with Multiple Experts](https://mlsanity.com/paper/2310.14774/) (2023)
- [Regression with reject option and application to kNN](https://mlsanity.com/paper/2006.16597/) (2020)
- [In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defer](https://mlsanity.com/paper/2311.01106/) (2023)

## Built on by (most cited first, 27 of 27)

- [Universal Model Routing for Efficient LLM Inference](https://mlsanity.com/paper/2502.08773/) (2025)
- [Regression with Multi-Expert Deferral](https://mlsanity.com/paper/2403.19494/) (2024)
- [Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defer](https://mlsanity.com/paper/2407.13732/) (2024)
- [Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction](https://mlsanity.com/paper/2501.00555/) (2024)
- [A Two-Stage Learning-to-Defer Approach for Multi-Task Learning](https://mlsanity.com/paper/2410.15729/) (2024)
- [Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defer](https://mlsanity.com/paper/2506.20650/) (2025)
- [Budgeted Multiple-Expert Deferral](https://mlsanity.com/paper/2510.26706/) (2025)
- [Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Experts](https://mlsanity.com/paper/2504.12988/) (2025)
- [A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problems](https://mlsanity.com/paper/2407.12710/) (2024)
- [Adversarial Robustness in One-Stage Learning-to-Defer](https://mlsanity.com/paper/2510.10988/) (2025)
- [Identity-Free Deferral For Unseen Experts](https://mlsanity.com/paper/2502.10533/) (2025)
- [Cost-Aware Routing for Efficient Text-To-Image Generation](https://mlsanity.com/paper/2506.14753/) (2025)
- [When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer](https://mlsanity.com/paper/2602.17144/) (2026)
- [Consistent Learning-to-Defer with Expert-Conditional Advice](https://mlsanity.com/paper/2603.14324/) (2026)
- [Fatigue-Aware Learning to Defer via Constrained Optimisation](https://mlsanity.com/paper/2604.00904/) (2026)
- [Learning to Help in Multi-Class Settings](https://mlsanity.com/paper/2501.13810/) (2025)
- [No Need for Learning to Defer? A Training Free Deferral Framework to Multiple Experts through Conformal Prediction](https://mlsanity.com/paper/2509.12573/) (2025)
- [L2CU: Learning to Complement Unseen Users](https://mlsanity.com/paper/2601.06119/) (2026)
- [Optimized Deferral for Imbalanced Settings](https://mlsanity.com/paper/2604.27723/) (2026)
- [From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascades](https://mlsanity.com/paper/2509.22984/) (2025)
- [Online Learning-to-Defer with Varying Experts](https://mlsanity.com/paper/2605.12340/) (2026)
- [Learning To Defer To A Population With Limited Demonstrations](https://mlsanity.com/paper/2510.19351/) (2025)
- [MPD2-Router: Mask-aware Multi-expert Prior-regularized Dual-head Deferral Router in Glaucoma Screening and Diagnosis](https://mlsanity.com/paper/2605.08024/) (2026)
- [Human-AI Teaming Through the Lens of Calibration](https://mlsanity.com/paper/2606.10906/) (2026)
- [From Few-Shot Segmentation to Clinician-in-the-Loop Medical Image Analysis](https://mlsanity.com/paper/2609.10001/) (2026)
- [RACER: Role-Aligned Competence Estimation for Human-AI Routing](https://mlsanity.com/paper/2609.21953/) (2026)
- [Incremental Retrieval-Augmented Generation for Human-in-the-Loop Decision Support](https://mlsanity.com/paper/2610.12666/) (2026)

## Abstract (arXiv)

The learning to defer (L2D) framework allows autonomous systems to be safe and robust by allocating difficult decisions to a human expert. All existing work on L2D assumes that each expert is well-identified, and if any expert were to change, the system should be re-trained. In this work, we alleviate this constraint, formulating an L2D system that can cope with never-before-seen experts at test-time. We accomplish this by using meta-learning, considering both optimization- and model-based variants. Given a small context set to characterize the currently available expert, our framework can quickly adapt its deferral policy. For the model-based approach, we employ an attention mechanism that is able to look for points in the context set that are similar to a given test point, leading to an even more precise assessment of the expert's abilities. In the experiments, we validate our methods on image recognition, traffic sign detection, and skin lesion diagnosis benchmarks.