mlsanityData to 7 Oct 2026

Learning to Complement and to Defer to Multiple Users

arXiv 2407.07003 ↗First posted 9 Jul 2024cs.CV cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

With the development of Human-AI Collaboration in Classification (HAI-CC), integrating users and AI predictions becomes challenging due to the complex decision-making process. This process has three options: 1) AI autonomously classifies, 2) learning to complement, where AI collaborates with users, and 3) learning to defer, where AI defers to users. Despite their interconnected nature, these options have been studied in isolation rather than as components of a unified system. In this paper, we address this weakness with the novel HAI-CC methodology, called Learning to Complement and to Defer to Multiple Users (LECODU). LECODU not only combines learning to complement and learning to defer strategies, but it also incorporates an estimation of the optimal number of users to engage in the decision process. The training of LECODU maximises classification accuracy and minimises collaboration costs associated with user involvement. Comprehensive evaluations across real-world and synthesized datasets demonstrate LECODU's superior performance compared to state-of-the-art HAI-CC methods. Remarkably, even when relying on unreliable users with high rates of label noise, LECODU exhibits significant improvement over both human decision-makers alone and AI alone.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 51 of its 78 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

51 of 78 references.

References · 51

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  7. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  8. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  9. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
  10. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  11. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  12. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
38 more
  1. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
  2. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 327
  3. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  4. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  5. 2019Human uncertainty makes classification more robustcited by 231
  6. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  7. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  8. 2020Learning to Complement Humanscited by 123
  9. 2017Deep learning from crowdscited by 105
  10. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
  11. 2020Multi-Objective Interpolation Training for Robustness to Label Noisecited by 86
  12. 2020Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noisecited by 82
  13. 2021Sample Selection with Uncertainty of Losses for Learning with Noisy Labelscited by 79
  14. 2017Who Said What: Modeling Individual Labelers Improves Classificationcited by 77
  15. 2021Differentiable Learning Under Triagecited by 74
  16. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 71
  17. 2021Learning Fast Sample Re-weighting Without Reward Datacited by 69
  18. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 68
  19. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 68
  20. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
  21. 2021Hard Sample Aware Noise Robust Learning for Histopathology Image Classificationcited by 48
  22. 2022ProMix: Combating Label Noise via Maximizing Clean Sample Utilitycited by 48
  23. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  24. 2021LongReMix: Robust Learning with High Confidence Samples in a Noisy Label Environmentcited by 40
  25. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  26. 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
  27. 2021Faster Meta Update Strategy for Noise-Robust Deep Learningcited by 38
  28. 2021Do Humans Trust Advice More if it Comes from AI? An Analysis of Human-AI Interactionscited by 35
  29. 2022Identifiability of Label Noise Transition Matrixcited by 34
  30. 2022On the Utility of Prediction Sets in Human-AI Teamscited by 34
  31. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  32. 2021Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possiblecited by 19
  33. 2022Instance-Dependent Noisy Label Learning via Graphical Modellingcited by 17
  34. 2018Learning Supervised Topic Models for Classification and Regression from Crowdscited by 16
  35. 2022CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotatorscited by 16
  36. 2021ScanMix: Learning from Severe Label Noise via Semantic Clustering and Semi-Supervised Learningcited by 13
  37. 2018Totally Looks Like - How Humans Compare, Compared to Machinescited by 12
  38. 2019Photometric Transformer Networks and Label Adjustment for Breast Density Predictioncited by 10

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Identity-Free Deferral For Unseen Expertscited by 4
  2. 2026Fatigue-Aware Learning to Defer via Constrained Optimisationcited by 3
  3. 2026L2CU: Learning to Complement Unseen Userscited by 2
  4. 2025DeCoDe: Defer-and-Complement Decision-Making via Decoupled Concept Bottleneck Modelscited by 0
  5. 2025Is Uncertainty Quantification a Viable Alternative to Learned Deferral?cited by 0
  6. 2025Learning To Defer To A Population With Limited Demonstrationscited by 0
  7. 2026People-Centred Medical Image Analysis via Fairness-Aware Human-AI Cooperationcited by 0