mlsanityData to 7 Oct 2026

A Causal Framework for Evaluating Deferring Systems

arXiv 2405.18902 ↗First posted 29 May 2024cs.LG cs.AI stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Deferring systems extend supervised Machine Learning (ML) models with the possibility to defer predictions to human experts. However, evaluating the impact of a deferring strategy on system accuracy is still an overlooked area. This paper fills this gap by evaluating deferring systems through a causal lens. We link the potential outcomes framework for causal inference with deferring systems, which allows to identify the causal impact of the deferring strategy on predictive accuracy. We distinguish two scenarios. In the first one, we have access to both the human and ML model predictions for the deferred instances. Here, we can identify the individual causal effects for deferred instances and the aggregates of them. In the second one, only human predictions are available for the deferred instances. Here, we can resort to regression discontinuity designs to estimate a local causal effect. We evaluate our approach on synthetic and real datasets for seven deferring systems from the literature.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 48 of its 133 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

48 of 133 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2016Wide Residual Networkscited by 3,057
  7. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  8. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  9. 2017Selective Classification for Deep Neural Networkscited by 799
  10. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  11. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  12. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
36 more
  1. 2015Strategic Classificationcited by 256
  2. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  3. 2019Addressing Failure Prediction by Learning Model Confidencecited by 183
  4. 2020Self-Adaptive Training: beyond Empirical Risk Minimizationcited by 126
  5. 2021Machine Learning with a Reject Option: A surveycited by 107
  6. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
  7. 2021Differentiable Learning Under Triagecited by 74
  8. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 68
  9. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 68
  10. 2021Optimal strategies for reject option classifierscited by 61
  11. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
  12. 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 57
  13. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  14. 2020Classification Under Human Assistancecited by 42
  15. 2019Hybrid Models with Deep and Invertible Featurescited by 41
  16. 2019Regression Under Human Assistancecited by 40
  17. 2021Role of Human-AI Interaction in Selective Predictioncited by 33
  18. 2023Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfactioncited by 32
  19. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  20. 2024Towards Human-AI Complementarity with Prediction Setscited by 25
  21. 2022Towards Better Selective Classificationcited by 21
  22. 2022AUC-based Selective Classificationcited by 13
  23. 2024Deep Neural Network Benchmarks for Selective Classificationcited by 12
  24. 2020Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policiescited by 9
  25. 2023Unsupervised Anomaly Detection with Rejectioncited by 9
  26. 2019Policy Evaluation with Latent Confounders via Optimal Balancecited by 8
  27. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  28. 2019A Practical Introduction to Regression Discontinuity Designs: Foundationscited by 6
  29. 2018Simple Local Polynomial Density Estimatorscited by 5
  30. 2021Regression Discontinuity Designscited by 5
  31. 2018Optimal Bandwidth Choice for Robust Bias Corrected Inference in Regression Discontinuity Designscited by 4
  32. 2019Capturing human categorization of natural images at scale by combining deep networks and cognitive modelscited by 3
  33. 2023A Guide to Regression Discontinuity Designs in Medical Applicationscited by 2
  34. 2023Counterfactually Comparing Abstaining Classifierscited by 2
  35. 2016Inference in Regression Discontinuity Designs with a Discrete Running Variablecited by 1
  36. 2018Extrapolating Treatment Effects in Multi-Cutoff Regression Discontinuity Designscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 6.

13 papers.

Sort:

By area:

  1. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  2. 2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 16
  3. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  4. 2025Budgeted Multiple-Expert Deferralcited by 11
  5. 2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 6
  6. 2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
  7. 2025To Ask or Not to Ask: Learning to Require Human Feedbackcited by 4
  8. 2026Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Modelscited by 4
  9. 2026When More Experts Hurt: Underfitting in Multi-Expert Learning to Defercited by 2
  10. 2026Consistent Learning-to-Defer with Expert-Conditional Advicecited by 1
  11. 2026Online Learning-to-Defer with Varying Expertscited by 0
  12. 2026One Human, N Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidencecited by 0
1 more
  1. 2026Should I stay or should I show? Learning to selectively disclose informationcited by 0