mlsanityData to 7 Oct 2026

Calibrated Learning to Defer with One-vs-All Classifiers

arXiv 2202.03673 ↗First posted 8 Feb 2022cs.LG stat.MLcited by 68

Not on the map. Why?

Abstract (arXiv)

The learning to defer (L2D) framework has the potential to make AI systems safer. For a given input, the system can defer the decision to a human if the human is more likely than the model to take the correct action. We study the calibration of L2D systems, investigating if the probabilities they output are sound. We find that Mozannar & Sontag's (2020) multiclass framework is not calibrated with respect to expert correctness. Moreover, it is not even guaranteed to produce valid probabilities due to its parameterization being degenerate for this purpose. We propose an L2D system based on one-vs-all classifiers that is able to produce calibrated probabilities of expert correctness. Furthermore, our loss function is also a consistent surrogate for multiclass L2D, like Mozannar & Sontag's (2020). Our experiments verify that not only is our system calibrated, but this benefit comes at no cost to accuracy. Our model's accuracy is always comparable (and often superior) to Mozannar & Sontag's (2020) model's in tasks ranging from hate speech detection to galaxy classification to diagnosis of skin lesions.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 24 of its 52 references that are in the map. First posted on arXiv 8 Feb 2022.

Builds on

24 of 52 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2016Wide Residual Networkscited by 3,057
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  7. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  8. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  9. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  10. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
  11. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  12. 2016FastText.zip: Compressing text classification modelscited by 331
12 more
  1. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  2. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  3. 2019Neural Legal Judgment Prediction in Englishcited by 208
  4. 2018To Trust Or Not To Trust A Classifiercited by 205
  5. 2019Evaluating model calibration in classificationcited by 165
  6. 2020Learning to Complement Humanscited by 123
  7. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
  8. 2021Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibrationcited by 79
  9. 2021Differentiable Learning Under Triagecited by 74
  10. 2020Classification with Rejection Based on Cost-sensitive Classificationcited by 55
  11. 2021Top-label calibration and multiclass-to-binary reductionscited by 46
  12. 2019On the Calibration of Multiclass Classification with Rejectioncited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 12; 2024: 17; 2025: 14; 2026: 23.

68 papers.

Sort:

By area:

  1. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  2. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 68
  3. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
  4. 2023Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithmscited by 36
  5. 2023Theoretically Grounded Loss Functions and Algorithms for Score-Based Multi-Class Abstentioncited by 32
  6. 2024Regression with Multi-Expert Deferralcited by 29
  7. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  8. 2024Learning to Defer to a Population: A Meta-Learning Approachcited by 26
  9. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  10. 2024A Universal Growth Rate for Learning with Smooth Surrogate Lossescited by 24
  11. 2024Enhanced H-Consistency Boundscited by 18
  12. 2022Human-AI Collaboration in Decision-Making: Beyond Learning to Defercited by 16
38 more
  1. 2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 16
  2. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  3. 2024Cascade-Aware Training of Language Modelscited by 15
  4. 2023Plugin estimators for selective classification with out-of-distribution detectioncited by 14
  5. 2024A Causal Framework for Evaluating Deferring Systemscited by 13
  6. 2024Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraintscited by 12
  7. 2025Budgeted Multiple-Expert Deferralcited by 11
  8. 2023Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Groupingcited by 10
  9. 2023Unified Classification and Rejection: A One-versus-All Frameworkcited by 9
  10. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  11. 2024Learning to Complement and to Defer to Multiple Userscited by 7
  12. 2025Human-AI Collaborative Uncertainty Quantificationcited by 6
  13. 2023Learning to Complement with Multiple Humanscited by 5
  14. 2024Online Decision Deferral under Budget Constraintscited by 5
  15. 2026When to Trust the Cheap Check: Weak and Strong Verification for Reasoningcited by 5
  16. 2026ReDAct: Uncertainty-Aware Deferral for LLM Agentscited by 5
  17. 2025Learning to Partially Defer for Sequencescited by 4
  18. 2025To Ask or Not to Ask: Learning to Require Human Feedbackcited by 4
  19. 2024Learning To Guide Human Decision Makers With Vision-Language Modelscited by 3
  20. 2026Fatigue-Aware Learning to Defer via Constrained Optimisationcited by 3
  21. 2023When No-Rejection Learning is Consistent for Regression with Rejectioncited by 2
  22. 2023Bayesian Online Learning for Consensus Predictioncited by 2
  23. 2024Diffusion Boosted Treescited by 2
  24. 2024Learning To Help: Training Models to Assist Legacy Devicescited by 2
  25. 2025Learning to Help in Multi-Class Settingscited by 2
  26. 2025Conformal Set-based Human-AI Complementarity with Multiple Expertscited by 2
  27. 2025No Need for Learning to Defer? A Training Free Deferral Framework to Multiple Experts through Conformal Predictioncited by 2
  28. 2026L2CU: Learning to Complement Unseen Userscited by 2
  29. 2026When More Experts Hurt: Underfitting in Multi-Expert Learning to Defercited by 2
  30. 2026Multi-Round Human-AI Collaboration with User-Specified Requirementscited by 2
  31. 2026Optimized Deferral for Imbalanced Settingscited by 2
  32. 2026TimeRouter: Efficient and Adaptive Routing of Time-Series Foundation Modelscited by 2
  33. 2024Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real Worldcited by 1
  34. 2025Bayesian Inference for Correlated Human Experts and Classifierscited by 1
  35. 2025MedGellan: LLM-Generated Medical Guidance to Support Physicianscited by 1
  36. 2025From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascadescited by 1
  37. 2026DeferredSeg:A Multi-Expert Deferral Framework for Medical Image Segmentationcited by 1
  38. 2026Calibrating conditional riskcited by 1

50 most cited of 68 All 68 (JSON)