# Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making arXiv: https://arxiv.org/abs/2208.07960 ยท HTML: https://mlsanity.com/paper/2208.07960/ ## Key facts - Cited by 33 papers in the map (data through 14 Oct 2026). - Builds on 13 of its 66 references that are in the map. - First posted on arXiv 16 Aug 2022. ## Builds on (most cited first, 13 of 13) - [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020) - [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020) - [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018) - [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018) - [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020) - [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020) - [Deep Learning for Identifying Metastatic Breast Cancer](https://mlsanity.com/paper/1606.05718/) (2016) - [A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores](https://mlsanity.com/paper/2002.08035/) (2020) - [Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making](https://mlsanity.com/paper/2010.07938/) (2020) - [What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play](https://mlsanity.com/paper/1810.09648/) (2018) - [Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging](https://mlsanity.com/paper/2205.09696/) (2022) - [Investigating Human + Machine Complementarity for Recidivism Predictions](https://mlsanity.com/paper/1808.09123/) (2018) - [Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA](https://mlsanity.com/paper/2012.15075/) (2020) ## Built on by (most cited first, 30 of 33) - [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023) - [Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence](https://mlsanity.com/paper/2404.00029/) (2024) - [Evaluating Human-AI Collaboration: A Review and Methodological Framework](https://mlsanity.com/paper/2407.19098/) (2024) - [Towards Human-AI Complementarity with Prediction Sets](https://mlsanity.com/paper/2405.17544/) (2024) - [The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment](https://mlsanity.com/paper/2412.16468/) (2024) - [Cost-Sensitive Learning to Defer to Multiple Experts with Workload Constraints](https://mlsanity.com/paper/2403.06906/) (2024) - [Generative AI in Knowledge Work: Design Implications for Data Navigation and Decision-Making](https://mlsanity.com/paper/2503.18419/) (2025) - [Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration](https://mlsanity.com/paper/2409.12809/) (2024) - [Towards Human-AI Complementarity in Matching Tasks](https://mlsanity.com/paper/2508.13285/) (2025) - [What's Producible May Not Be Reachable: Measuring the Steerability of Generative Models](https://mlsanity.com/paper/2503.17482/) (2025) - [How Managers Perceive AI-Assisted Conversational Training for Workplace Communication](https://mlsanity.com/paper/2505.14452/) (2025) - [Epistemology gives a Future to Complementarity in Human-AI Interactions](https://mlsanity.com/paper/2601.09871/) (2026) - [FiFAR: A Fraud Detection Dataset for Learning to Defer](https://mlsanity.com/paper/2312.13218/) (2023) - [Cognitive Spillover in Human-AI Teams](https://mlsanity.com/paper/2407.17489/) (2024) - [Mental Models of Autonomy and Sentience Shape Reactions to AI](https://mlsanity.com/paper/2512.09085/) (2025) - [Towards Human-AI Mutual Learning: A New Research Paradigm](https://mlsanity.com/paper/2405.04687/) (2024) - [Does AI Coaching Prepare us for Workplace Negotiations?](https://mlsanity.com/paper/2509.22545/) (2025) - [AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science](https://mlsanity.com/paper/2603.19005/) (2026) - [A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health](https://mlsanity.com/paper/2411.02594/) (2024) - [Achieving Trustworthy Real-Time Decision Support Systems with Low-Latency Interpretable AI Models](https://mlsanity.com/paper/2506.20018/) (2025) - [Is Passive Expertise-Based Personalization Enough? A Case Study in AI-Assisted Test-Taking](https://mlsanity.com/paper/2511.23376/) (2025) - [AI as Teammate or Tool? A Review of Human-AI Interaction in Decision Support](https://mlsanity.com/paper/2602.15865/) (2026) - [Towards an Appropriate Level of Reliance on AI: A Preliminary Reliance-Control Framework for AI in Software Engineering](https://mlsanity.com/paper/2604.10530/) (2026) - [Enabling Collaborative Clinical Diagnosis of Infectious Keratitis by Integrating Expert Knowledge and Interpretable Data-driven Intelligence](https://mlsanity.com/paper/2401.08695/) (2024) - [What Do People Want to Know About Artificial Intelligence (AI)? The Importance of Answering End-User Questions to Explain Autonomous Vehicle (AV) Decisions](https://mlsanity.com/paper/2505.06428/) (2025) - [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026) - [Human-AI Collaboration in Large Language Model-Integrated Building Energy Management Systems: The Role of User Domain Knowledge and AI Literacy](https://mlsanity.com/paper/2602.16140/) (2026) - [Dynamic Agentic AI Expert Profiler System Architecture for Multidomain Intelligence Modeling](https://mlsanity.com/paper/2604.05345/) (2026) - [Centering Ecological Goals in Automated Identification of Individual Animals](https://mlsanity.com/paper/2604.20626/) (2026) - [Addressing the Synergy Gap: The Six Elements of the Design Space](https://mlsanity.com/paper/2605.21635/) (2026) ## Abstract (arXiv) Human-AI collaboration for decision-making strives to achieve team performance that exceeds the performance of humans or AI alone. However, many factors can impact success of Human-AI teams, including a user's domain expertise, mental models of an AI system, trust in recommendations, and more. This work examines users' interaction with three simulated algorithmic models, all with similar accuracy but different tuning on their true positive and true negative rates. Our study examined user performance in a non-trivial blood vessel labeling task where participants indicated whether a given blood vessel was flowing or stalled. Our results show that while recommendations from an AI-Assistant can aid user decision making, factors such as users' baseline performance relative to the AI and complementary tuning of AI error types significantly impact overall team performance. Novice users improved, but not to the accuracy level of the AI. Highly proficient users were generally able to discern when they should follow the AI recommendation and typically maintained or improved their performance. Mid-performers, who had a similar level of accuracy to the AI, were most variable in terms of whether the AI recommendations helped or hurt their performance. In addition, we found that users' perception of the AI's performance relative on their own also had a significant impact on whether their accuracy improved when given AI recommendations. This work provides insights on the complexity of factors related to Human-AI collaboration and provides recommendations on how to develop human-centered AI algorithms to complement users in decision-making tasks.