mlsanityData to 14 Oct 2026

Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defer

arXiv 2407.13732 ↗First posted 18 Jul 2024cs.LG stat.MLcited by 26

Not on the map. Why?

Abstract (arXiv)

We present a comprehensive study of surrogate loss functions for learning to defer. We introduce a broad family of surrogate losses, parameterized by a non-increasing function Psi, and establish their realizable H-consistency under mild conditions. For cost functions based on classification error, we further show that these losses admit H-consistency bounds when the hypothesis set is symmetric and complete, a property satisfied by common neural network and linear function hypothesis sets. Our results also resolve an open question raised in previous work (Mozannar et al., 2023) by proving the realizable H-consistency and Bayes-consistency of a specific surrogate loss. Furthermore, we identify choices of Psi that lead to H-consistent surrogate losses for any general cost function, thus achieving Bayes-consistency, realizable H-consistency, and H-consistency bounds simultaneously. We also investigate the relationship between H-consistency bounds and realizable H-consistency in learning to defer, highlighting key differences from standard classification. Finally, we empirically evaluate our proposed surrogate losses and compare them with existing baselines.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 56 of its 107 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

56 of 107 references.

References · 56

  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  4. 2017Selective Classification for Deep Neural Networkscited by 852
  5. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  6. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
  7. 2018Learning Adversarially Fair and Transferable Representationscited by 353
  8. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  9. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  10. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 210
  11. 2020Learning to Complement Humanscited by 124
  12. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
38 more
  1. 2021Differentiable Learning Under Triagecited by 76
  2. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 72
  3. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 70
  4. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 69
  5. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 60
  6. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
  7. 2020Classification with Rejection Based on Cost-sensitive Classificationcited by 55
  8. 2021Calibration and Consistency of Adversarial Surrogate Lossescited by 47
  9. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  10. 2020Classification Under Human Assistancecited by 42
  11. 2019On the Calibration of Multiclass Classification with Rejectioncited by 41
  12. 2019Regression Under Human Assistancecited by 40
  13. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  14. 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
  15. 2020Selective Classification via One-Sided Predictioncited by 38
  16. 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
  17. 2021Human-AI Collaboration with Bandit Feedbackcited by 36
  18. 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
  19. 2021Selective Regression Under Fairness Criteriacited by 34
  20. 2023Learning to Reject with a Fixed Predictor: Application to Decontextualizationcited by 32
  21. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  22. 2020Regression with reject option and application to kNNcited by 30
  23. 2021Teaching Humans When To Defer to a Classifier via Exemplarscited by 29
  24. 2023Revisiting Discriminative vs. Generative Classifiers: Theory and Implicationscited by 29
  25. 2024Regression with Multi-Expert Deferralcited by 29
  26. 2023Ranking with Abstentioncited by 27
  27. 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 27
  28. 2021A Finer Calibration Analysis for Adversarial Robustnesscited by 26
  29. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  30. 2023Learning to Defer with Limited Expert Predictionscited by 25
  31. 2024A Universal Growth Rate for Learning with Smooth Surrogate Lossescited by 24
  32. 2024Cardinality-Aware Set Prediction and Top-k Classificationcited by 23
  33. 2024H-Consistency Guarantees for Regressioncited by 21
  34. 2021Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possiblecited by 19
  35. 2021Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosiscited by 17
  36. 2021Reinforcement Learning Under Algorithmic Triagecited by 15
  37. 2022Counterfactual Inference of Second Opinionscited by 14
  38. 2023Learning to Make Adherence-Aware Advicecited by 13

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 10.

26 papers.

Sort:

By area:

  1. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  2. 2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 17
  3. 2025Adversarial Robustness in Two-Stage Learning-to-Defer: Algorithms and Guaranteescited by 16
  4. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  5. 2024Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guaranteescited by 15
  6. 2025Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Datacited by 15
  7. 2025Budgeted Multiple-Expert Deferralcited by 13
  8. 2025Improved Balanced Classification with Theoretically Grounded Loss Functionscited by 9
  9. 2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 7
  10. 2025Beyond Tsybakov: Model Margin Noise and H-Consistency Boundscited by 7
  11. 2025To Ask or Not to Ask: Learning to Require Human Feedbackcited by 6
  12. 2025Principled Algorithms for Optimizing Generalized Metrics in Binary Classificationcited by 6
14 more
  1. 2026Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Modelscited by 6
  2. 2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
  3. 2026A Theoretical Framework for Modular Learning of Robust Generative Modelscited by 4
  4. 2026When More Experts Hurt: Underfitting in Multi-Expert Learning to Defercited by 3
  5. 2026Optimized Deferral for Imbalanced Settingscited by 2
  6. 2026Linear-Core Surrogates: Smooth Loss Functions with Linear Rates for Classification and Structured Predictioncited by 2
  7. 2025Contrasting Cost-Agnostic and Cost-Sensitive Losses under Limited Model Capacity via mathcal H-consistencycited by 1
  8. 2025A Connection Between Learning to Reject and Bhattacharyya Divergencescited by 1
  9. 2026Online Learning-to-Defer with Varying Expertscited by 1
  10. 2025Is Uncertainty Quantification a Viable Alternative to Learned Deferral?cited by 0
  11. 2026Dissecting Model Failures in Abdominal Aortic Aneurysm Segmentation through Explainability-Driven Analysiscited by 0
  12. 2026Principled Algorithms for Optimizing Generalized Metrics in Multi-Label Learningcited by 0
  13. 2026Human-AI Teaming Through the Lens of Calibrationcited by 0
  14. 2026Verifiable abstention makes AI leak diagnosis accountable in urban water distribution networkscited by 0