mlsanityData to 14 Oct 2026

Bayesian Inference for Correlated Human Experts and Classifiers

arXiv 2506.05636 ↗First posted 5 Jun 2025cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Applications of machine learning often involve making predictions based on both model outputs and the opinions of human experts. In this context, we investigate the problem of querying experts for class label predictions, using as few human queries as possible, and leveraging the class probability estimates of pre-trained classifiers. We develop a general Bayesian framework for this problem, modeling expert correlation via a joint latent representation, enabling simulation-based inference about the utility of additional expert queries, as well as inference of posterior distributions over unobserved expert labels. We apply our approach to two real-world medical classification problems, as well as to CIFAR-10H and ImageNet-16H, demonstrating substantial reductions relative to baselines in the cost of querying human experts while maintaining high prediction accuracy.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

33 of 59 references.

References · 33

  1. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  2. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
  3. 2019Verified Uncertainty Calibrationcited by 309
  4. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  5. 2019Human uncertainty makes classification more robustcited by 235
  6. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  7. 2023Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanationscited by 83
  8. 2020A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scorescited by 74
  9. 2021Towards Unbiased and Accurate Deferral to Multiple Expertscited by 72
  10. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 70
  11. 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 69
  12. 2023Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Makingcited by 67
21 more
  1. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 60
  2. 2021Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Makingcited by 58
  3. 2022Improving Expert Predictions with Conformal Predictioncited by 54
  4. 2017How hard can it be? Estimating the difficulty of visual search in an imagecited by 52
  5. 2021Hard Sample Aware Noise Robust Learning for Histopathology Image Classificationcited by 48
  6. 2022Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairnesscited by 45
  7. 2021Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibrationcited by 40
  8. 2022Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Expertscited by 40
  9. 2023Human-Aligned Calibration for AI-Assisted Decision Makingcited by 28
  10. 2023Learning to Defer with Limited Expert Predictionscited by 25
  11. 2022A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementaritycited by 23
  12. 2022Learning When to Advise Human Decision Makerscited by 18
  13. 2023Learning Personalized Decision Support Policiescited by 14
  14. 2020Deep Learning for Distinguishing Normal versus Abnormal Chest Radiographs and Generalization to Unseen Diseasescited by 12
  15. 2021Improving Learning-to-Defer Algorithms Through Fine-Tuningcited by 11
  16. 2023Learning Complementary Policies for Human-AI Teamscited by 7
  17. 2025Bayesian Optimization for Building Social-Influence-Free Consensuscited by 6
  18. 2022A Cognitive Framework for Delegation Between Error-Prone AI and Human Agentscited by 5
  19. 2023Online Decision Mediationcited by 4
  20. 2021Bayesian Classifier Fusion with an Explicit Model of Correlationcited by 2
  21. 2023Bayesian Online Learning for Consensus Predictioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Human-AI Teaming Through the Lens of Calibrationcited by 0