# Learning To Defer To A Population With Limited Demonstrations

arXiv: https://arxiv.org/abs/2510.19351 · HTML: https://mlsanity.com/paper/2510.19351/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Builds on (most cited first, 19 of 19)

- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019)
- [Differentiable Learning Under Triage](https://mlsanity.com/paper/2103.08902/) (2021)
- [Calibrated Learning to Defer with One-vs-All Classifiers](https://mlsanity.com/paper/2202.03673/) (2022)
- [Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles](https://mlsanity.com/paper/2210.16955/) (2022)
- [Hybrid Intelligence](https://mlsanity.com/paper/2105.00691/) (2021)
- [Regression Under Human Assistance](https://mlsanity.com/paper/1909.02963/) (2019)
- [Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts](https://mlsanity.com/paper/2206.07948/) (2022)
- [Human-AI Collaboration with Bandit Feedback](https://mlsanity.com/paper/2105.10614/) (2021)
- [Learning to Defer to a Population: A Meta-Learning Approach](https://mlsanity.com/paper/2403.02683/) (2024)
- [Learning to Defer with Limited Expert Predictions](https://mlsanity.com/paper/2304.07306/) (2023)
- [Improving Learning-to-Defer Algorithms Through Fine-Tuning](https://mlsanity.com/paper/2112.10768/) (2021)
- [CLOC: Contrastive Learning for Ordinal Classification with Multi-Margin N-pair Loss](https://mlsanity.com/paper/2504.17813/) (2025)
- [Learning to Complement and to Defer to Multiple Users](https://mlsanity.com/paper/2407.07003/) (2024)
- [Identity-Free Deferral For Unseen Experts](https://mlsanity.com/paper/2502.10533/) (2025)

## Abstract (arXiv)

This paper addresses the critical data scarcity that hinders the practical deployment of learning to defer (L2D) systems to the population. We introduce a context-aware, semi-supervised framework that uses meta-learning to generate expert-specific embeddings from only a few demonstrations. We demonstrate the efficacy of a dual-purpose mechanism, where these embeddings are used first to generate a large corpus of pseudo-labels for training, and subsequently to enable on-the-fly adaptation to new experts at test-time. The experiment results on three different datasets confirm that a model trained on these synthetic labels rapidly approaches oracle-level performance, validating the data efficiency of our approach. By resolving a key training bottleneck, this work makes adaptive L2D systems more practical and scalable, paving the way for human-AI collaboration in real-world environments. To facilitate reproducibility and address implementation details not covered in the main text, we provide our source code and training configurations at https://github.com/nil123532/learning-to-defer-to-a-population-with-limited-demonstrations.