mlsanityData to 7 Oct 2026

Designing Closed Human-in-the-loop Deferral Pipelines

arXiv 2202.04718 ↗First posted 9 Feb 2022cs.HC cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

In hybrid human-machine deferral frameworks, a classifier can defer uncertain cases to human decision-makers (who are often themselves fallible). Prior work on simultaneous training of such classifier and deferral models has typically assumed access to an oracle during training to obtain true class labels for training samples, but in practice there often is no such oracle. In contrast, we consider a "closed" decision-making pipeline in which the same fallible human decision-makers used in deferral also provide training labels. How can imperfect and biased human expert labels be used to train a fair and accurate deferral framework? Our key insight is that by exploiting weak prior information, we can match experts to input examples to ensure fairness and accuracy of the resulting deferral framework, even when imperfect and biased experts are used in place of ground truth labels. The efficacy of our approach is shown both by theoretical analysis and by evaluation on two tasks.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 16 of its 49 references that are in the map. First posted on arXiv 9 Feb 2022.

Builds on

16 of 49 references.

References · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  3. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  4. 2017A Tutorial on Thompson Samplingcited by 420
  5. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  6. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  7. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  8. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  9. 2017Runaway Feedback Loops in Predictive Policingcited by 97
  10. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 71
  11. 2019Deep Gamblers: Learning to Abstain with Portfolio Theorycited by 69
  12. 2018Direct Uncertainty Prediction for Medical Second Opinionscited by 61
4 more
  1. 2018Mitigating Bias in Adaptive Data Gathering via Differential Privacycited by 21
  2. 2018Enhancing the Accuracy and Fairness of Human Decision Makingcited by 13
  3. 2018Crowdsourcing with Fairness, Diversity and Budget Constraintscited by 9
  4. 2021Auditing for Diversity using Representative Examplescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 3; 2025: 2; 2026: 1.

11 papers.

  1. 2022Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairnesscited by 45
  2. 2022Human-AI Collaboration in Decision-Making: Beyond Learning to Defercited by 16
  3. 2024Human Expertise in Algorithmic Predictioncited by 13
  4. 2025Explaining and Improving Information Complementarities in Multi-Agent Decision-makingcited by 6
  5. 2023Inclusive Portraits: Race-Aware Human-in-the-Loop Technologycited by 3
  6. 2024Learning To Guide Human Decision Makers With Vision-Language Modelscited by 3
  7. 2024Integrating Expert Judgment and Algorithmic Decision Making: An Indistinguishability Frameworkcited by 3
  8. 2023Improving the Efficiency of Human-in-the-Loop Systems: Adding Artificial to Human Expertscited by 1
  9. 2025MedGellan: LLM-Generated Medical Guidance to Support Physicianscited by 1
  10. 2026ComplLLM: Fine-tuning LLMs to Discover Complementary Signals for Decision-makingcited by 1
  11. 2023Learning to Guide Human Experts via Personalized Large Language Modelscited by 0