# The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems

arXiv: https://arxiv.org/abs/2105.03354 · HTML: https://mlsanity.com/paper/2105.03354/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 84 references that are in the map.
- First posted on arXiv 7 May 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification](https://mlsanity.com/paper/1503.01161/) (2015)
- [Machine Teaching: A New Paradigm for Building Machine Learning Systems](https://mlsanity.com/paper/1707.06742/) (2017)
- [Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems](https://mlsanity.com/paper/1712.01785/) (2017)
- [On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems](https://mlsanity.com/paper/1611.08309/) (2016)
- [Debugging Machine Learning Tasks](https://mlsanity.com/paper/1603.07292/) (2016)
- [Crowdsourcing Feature Discovery via Adaptively Chosen Comparisons](https://mlsanity.com/paper/1504.00064/) (2015)
- [An Active Learning Approach for Jointly Estimating Worker Performance and Annotation Reliability with Crowdsourced Data](https://mlsanity.com/paper/1401.3836/) (2014)
- [Interpretable and Pedagogical Examples](https://mlsanity.com/paper/1711.00694/) (2017)
- [Co-active Learning to Adapt Humanoid Movement for Manipulation](https://mlsanity.com/paper/1609.03628/) (2016)

## Built on by (most cited first, 30 of 39)

- [Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI](https://mlsanity.com/paper/2105.05424/) (2021)
- [Evaluating Human-AI Collaboration: A Review and Methodological Framework](https://mlsanity.com/paper/2407.19098/) (2024)
- [Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience](https://mlsanity.com/paper/2302.10395/) (2023)
- [AI-enhanced Collective Intelligence](https://mlsanity.com/paper/2403.10433/) (2024)
- [A Test for Evaluating Performance in Human-Computer Systems](https://mlsanity.com/paper/2206.12390/) (2022)
- [Conversational AI-Powered Design: ChatGPT as Designer, User, and Product](https://mlsanity.com/paper/2302.07406/) (2023)
- [Unpacking Human-AI interactions: From interaction primitives to a design space](https://mlsanity.com/paper/2401.05115/) (2024)
- [Interrogating AI: Characterizing Emergent Playful Interactions with ChatGPT](https://mlsanity.com/paper/2401.08405/) (2024)
- ["What's Up, Doc?": Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets](https://mlsanity.com/paper/2506.21532/) (2025)
- [Painting the black box white: experimental findings from applying XAI to an ECG reading setting](https://mlsanity.com/paper/2210.15236/) (2022)
- [Monotonicity for AI ethics and society: An empirical study of the monotonic neural additive model in criminology, education, health care, and finance](https://mlsanity.com/paper/2301.07060/) (2023)
- [IMAD: IMage-Augmented multi-modal Dialogue](https://mlsanity.com/paper/2305.10512/) (2023)
- [On the Ethical Considerations of Generative Agents](https://mlsanity.com/paper/2411.19211/) (2024)
- [Artificial intelligence in government: Concepts, standards, and a unified framework](https://mlsanity.com/paper/2210.17218/) (2022)
- [Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2505.21907/) (2025)
- [The Epistemological Consequences of Large Language Models: Rethinking collective intelligence and institutional knowledge](https://mlsanity.com/paper/2512.19570/) (2025)
- [Modeling Human Behavior Part I -- Learning and Belief Approaches](https://mlsanity.com/paper/2205.06485/) (2022)
- [Rethinking and Recomputing the Value of Machine Learning Models](https://mlsanity.com/paper/2209.15157/) (2022)
- [Trustworthy Human Computation: A Survey](https://mlsanity.com/paper/2210.12324/) (2022)
- [Improving the Efficiency of Human-in-the-Loop Systems: Adding Artificial to Human Experts](https://mlsanity.com/paper/2307.03003/) (2023)
- [The two-way knowledge interaction interface between humans and neural networks](https://mlsanity.com/paper/2401.05461/) (2024)
- [Eliciting Model Steering Interactions from Users via Data and Visual Design Probes](https://mlsanity.com/paper/2310.09314/) (2023)
- [Human-Generative AI Collaborative Problem Solving Who Leads and How Students Perceive the Interactions](https://mlsanity.com/paper/2405.13048/) (2024)
- [A Beautiful Mind: Principles and Strategies for AI-Augmented Human Reasoning](https://mlsanity.com/paper/2503.15530/) (2025)
- [A Human Digital Twin Architecture for Knowledge-based Interactions and Context-Aware Conversations](https://mlsanity.com/paper/2504.03147/) (2025)
- [Human aversion? Do AI Agents Judge Identity More Harshly Than Performance](https://mlsanity.com/paper/2504.13871/) (2025)
- [Human-Centered Human-AI Collaboration (HCHAC)](https://mlsanity.com/paper/2505.22477/) (2025)
- [Exploring Human-AI Collaboration Using Mental Models of Early Adopters of Multi-Agent Generative AI Tools](https://mlsanity.com/paper/2510.06224/) (2025)
- [The Rise of the Knowledge Sculptor: A New Archetype for Knowledge Work in the Age of Generative AI](https://mlsanity.com/paper/2510.07829/) (2025)
- [Evaluating Cultural Knowledge Processing in Large Language Models: A Cognitive Benchmarking Framework Integrating Retrieval-Augmented Generation](https://mlsanity.com/paper/2511.01649/) (2025)

## Abstract (arXiv)

Recent technological advances, especially in the field of machine learning, provide astonishing progress on the road towards artificial general intelligence. However, tasks in current real-world business applications cannot yet be solved by machines alone. We, therefore, identify the need for developing socio-technological ensembles of humans and machines. Such systems possess the ability to accomplish complex goals by combining human and artificial intelligence to collectively achieve superior results and continuously improve by learning from each other. Thus, the need for structured design knowledge for those systems arises. Following a taxonomy development method, this article provides three main contributions: First, we present a structured overview of interdisciplinary research on the role of humans in the machine learning pipeline. Second, we envision hybrid intelligence systems and conceptualize the relevant dimensions for system design for the first time. Finally, we offer useful guidance for system developers during the implementation of such applications.