# A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementarity

arXiv: https://arxiv.org/abs/2204.10806 · HTML: https://mlsanity.com/paper/2204.10806/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 83 references that are in the map.
- First posted on arXiv 22 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices](https://mlsanity.com/paper/1906.09208/) (2019)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [Optimal Sparse Decision Trees](https://mlsanity.com/paper/1904.12847/) (2019)
- [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Differentiable Learning Under Triage](https://mlsanity.com/paper/2103.08902/) (2021)
- [A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores](https://mlsanity.com/paper/2002.08035/) (2020)
- [Towards Unbiased and Accurate Deferral to Multiple Experts](https://mlsanity.com/paper/2102.13004/) (2021)
- [Adversarial Attacks Against Medical Deep Learning Systems](https://mlsanity.com/paper/1804.05296/) (2018)
- [Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making](https://mlsanity.com/paper/2010.07938/) (2020)
- [Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness](https://mlsanity.com/paper/2202.08821/) (2022)
- [Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Support](https://mlsanity.com/paper/2204.02310/) (2022)
- [Human-AI Collaboration with Bandit Feedback](https://mlsanity.com/paper/2105.10614/) (2021)
- [A Human-Centered Review of the Algorithms used within the U.S. Child Welfare System](https://mlsanity.com/paper/2003.03541/) (2020)
- [Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government Contexts](https://mlsanity.com/paper/2012.05370/) (2020)
- [Investigating Human + Machine Complementarity for Recidivism Predictions](https://mlsanity.com/paper/1808.09123/) (2018)
- [Designing for human-AI complementarity in K-12 education](https://mlsanity.com/paper/2104.01266/) (2021)
- [The Impact of Algorithmic Risk Assessments on Human Predictions and its Analysis via Crowdsourcing Studies](https://mlsanity.com/paper/2109.01443/) (2021)

## Built on by (most cited first, 23 of 23)

- [When combinations of humans and AI are useful: A systematic review and meta-analysis](https://mlsanity.com/paper/2405.06087/) (2024)
- [Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence](https://mlsanity.com/paper/2404.00029/) (2024)
- [Human Expertise in Algorithmic Prediction](https://mlsanity.com/paper/2402.00793/) (2024)
- [Predicting and Understanding Human Action Decisions: Insights from Large Language Models and Cognitive Instance-Based Learning](https://mlsanity.com/paper/2407.09281/) (2024)
- [Explaining and Improving Information Complementarities in Multi-Agent Decision-making](https://mlsanity.com/paper/2502.06152/) (2025)
- [Learning to Complement with Multiple Humans](https://mlsanity.com/paper/2311.13172/) (2023)
- [Epistemology gives a Future to Complementarity in Human-AI Interactions](https://mlsanity.com/paper/2601.09871/) (2026)
- [Integrating Expert Judgment and Algorithmic Decision Making: An Indistinguishability Framework](https://mlsanity.com/paper/2410.08783/) (2024)
- [Multi-Round Human-AI Collaboration with User-Specified Requirements](https://mlsanity.com/paper/2602.17646/) (2026)
- [Collaborative Causal Sensemaking: Closing the Complementarity Gap in Human-AI Decision Support](https://mlsanity.com/paper/2512.07801/) (2025)
- [ComplLLM: Fine-tuning LLMs to Discover Complementary Signals for Decision-making](https://mlsanity.com/paper/2602.19458/) (2026)
- [Optimising Human-Machine Collaboration for Efficient High-Precision Information Extraction from Text Documents](https://mlsanity.com/paper/2302.09324/) (2023)
- [Bayesian Inference for Correlated Human Experts and Classifiers](https://mlsanity.com/paper/2506.05636/) (2025)
- [Strategic Decision Support for AI Agents](https://mlsanity.com/paper/2606.12587/) (2026)
- [Align AI to Dynamic Human-AI Workflows](https://mlsanity.com/paper/2607.14240/) (2026)
- [AI-assisted Gaze Detection for Proctoring Online Exams](https://mlsanity.com/paper/2409.16923/) (2024)
- [The safe harbor paradox: when combining human and AI skills creates or destroys value](https://mlsanity.com/paper/2509.14057/) (2025)
- [Development of Mental Models in Human-AI Collaboration: A Conceptual Framework](https://mlsanity.com/paper/2510.08104/) (2025)
- [PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AI](https://mlsanity.com/paper/2605.05682/) (2026)
- [From Model Uncertainty to Human Attention: Localization-Aware Visual Cues for Scalable Annotation Review](https://mlsanity.com/paper/2605.12303/) (2026)
- [Addressing the Synergy Gap: The Six Elements of the Design Space](https://mlsanity.com/paper/2605.21635/) (2026)
- [Tree-Based Formalization of Multi-Agent Complementarity in Human-AI Interactions](https://mlsanity.com/paper/2606.04779/) (2026)
- [LLM Detection as an Intervention: Downstream Impact under Strategic User Behavior](https://mlsanity.com/paper/2607.19300/) (2026)

## Abstract (arXiv)

Hybrid human-ML systems increasingly make consequential decisions in a wide range of domains. These systems are often introduced with the expectation that the combined human-ML system will achieve complementary performance, that is, the combined decision-making system will be an improvement compared with either decision-making agent in isolation. However, empirical results have been mixed, and existing research rarely articulates the sources and mechanisms by which complementary performance is expected to arise. Our goal in this work is to provide conceptual tools to advance the way researchers reason and communicate about human-ML complementarity. Drawing upon prior literature in human psychology, machine learning, and human-computer interaction, we propose a taxonomy characterizing distinct ways in which human and ML-based decision-making can differ. In doing so, we conceptually map potential mechanisms by which combining human and ML decision-making may yield complementary performance, developing a language for the research community to reason about design of hybrid systems in any decision-making domain. To illustrate how our taxonomy can be used to investigate complementarity, we provide a mathematical aggregation framework to examine enabling conditions for complementarity. Through synthetic simulations, we demonstrate how this framework can be used to explore specific aspects of our taxonomy and shed light on the optimal mechanisms for combining human-ML judgments