mlsanityData to 7 Oct 2026

Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration

arXiv 2109.14591 ↗First posted 29 Sep 2021cs.LG stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

An increasingly common use case for machine learning models is augmenting the abilities of human decision makers. For classification tasks where neither the human or model are perfectly accurate, a key step in obtaining high performance is combining their individual predictions in a manner that leverages their relative strengths. In this work, we develop a set of algorithms that combine the probabilistic output of a model with the class-level output of a human. We show theoretically that the accuracy of our combination model is driven not only by the individual human and model accuracies, but also by the model's confidence. Empirical results on image classification with CIFAR-10 and a subset of ImageNet demonstrate that such human-model combinations consistently have higher accuracies than the model or human alone, and that the parameters of the combination method can be estimated effectively with as few as ten labeled datapoints.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 25 of its 69 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

25 of 69 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  8. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  9. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  10. 2019Verified Uncertainty Calibrationcited by 307
  11. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  12. 2019Human uncertainty makes classification more robustcited by 231
13 more
  1. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  2. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  3. 2020Learning to Complement Humanscited by 123
  4. 2019The Algorithmic Automation Problem: Prediction, Triage, and Human Effortcited by 101
  5. 2020Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistencycited by 77
  6. 2021Differentiable Learning Under Triagecited by 74
  7. 2017Conditional independence testing based on a nearest-neighbor estimator of conditional mutual informationcited by 70
  8. 2019Human-Centered Artificial Intelligence and Machine Learningcited by 37
  9. 2019CCMI : Classifier based Conditional Mutual Information Estimationcited by 37
  10. 2020Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binningcited by 27
  11. 2018Totally Looks Like - How Humans Compare, Compared to Machinescited by 12
  12. 2019Testing Conditional Independence on Discrete Data using Stochastic Complexitycited by 10
  13. 2021Human-AI Symbiosis: A Survey of Current Approachescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 8; 2024: 9; 2025: 10; 2026: 7.

40 papers.

Sort:

  1. 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 68
  2. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
  3. 2022Improving Expert Predictions with Conformal Predictioncited by 54
  4. 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
  5. 2022Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairnesscited by 45
  6. 2023Human Uncertainty in Concept-Based AI Systemscited by 39
  7. 2023Principled Approaches for Learning to Defer with Multiple Expertscited by 38
  8. 2023Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfactioncited by 32
  9. 2024Regression with Multi-Expert Deferralcited by 29
  10. 2023In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defercited by 26
  11. 2024Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defercited by 26
  12. 2024A Two-Stage Learning-to-Defer Approach for Multi-Task Learningcited by 16
28 more
  1. 2025Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defercited by 16
  2. 2022Counterfactual Inference of Second Opinionscited by 14
  3. 2024Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?cited by 13
  4. 2025Budgeted Multiple-Expert Deferralcited by 11
  5. 2024Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scarycited by 9
  6. 2025From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysiscited by 9
  7. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  8. 2024Learning to Complement and to Defer to Multiple Userscited by 7
  9. 2025Why Ask One When You Can Ask k? Learning-to-Defer to the Top-k Expertscited by 6
  10. 2023Learning to Complement with Multiple Humanscited by 5
  11. 2025Adversarial Robustness in One-Stage Learning-to-Defercited by 5
  12. 2022Factors that influence the adoption of human-AI collaboration in clinical decision-makingcited by 4
  13. 2025How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyondcited by 4
  14. 2025Narrowing Action Choices with AI Improves Human Sequential Decisionscited by 3
  15. 2023Bayesian Online Learning for Consensus Predictioncited by 2
  16. 2025Learning to Help in Multi-Class Settingscited by 2
  17. 2025Conformal Set-based Human-AI Complementarity with Multiple Expertscited by 2
  18. 2026L2CU: Learning to Complement Unseen Userscited by 2
  19. 2024A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessmentcited by 1
  20. 2024Human-AI Collaborative Multi-modal Multi-rater Learning for Endometriosis Diagnosiscited by 1
  21. 2025Bayesian Inference for Correlated Human Experts and Classifierscited by 1
  22. 2026Consistent Learning-to-Defer with Expert-Conditional Advicecited by 1
  23. 2023Uncertainty Estimation based on Geometric Separationcited by 0
  24. 2026People-Centred Medical Image Analysis via Fairness-Aware Human-AI Cooperationcited by 0
  25. 2026PLACO: A Multi-Stage Framework for Cost-Effective Performance in Human-AI Teamscited by 0
  26. 2026Online Learning-to-Defer with Varying Expertscited by 0
  27. 2026Human-AI Teaming Through the Lens of Calibrationcited by 0
  28. 2026CORDIAL: Calibrating Ordinal LLM Outputs from Few Labelscited by 0