# Toward Human-AI Complementarity Across Diverse Tasks

arXiv: https://arxiv.org/abs/2605.04070 · HTML: https://mlsanity.com/paper/2605.04070/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Builds on (most cited first, 30 of 30)

- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [Supervising strong learners by amplifying weak experts](https://mlsanity.com/paper/1810.08575/) (2018)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022)
- [Question Decomposition Improves the Faithfulness of Model-Generated Reasoning](https://mlsanity.com/paper/2307.11768/) (2023)
- [Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making](https://mlsanity.com/paper/2010.07938/) (2020)
- [SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge](https://mlsanity.com/paper/2509.07968/) (2025)
- [Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies](https://mlsanity.com/paper/2502.08554/) (2025)
- [Role of Human-AI Interaction in Selective Prediction](https://mlsanity.com/paper/2112.06751/) (2021)
- [Scaling Laws For Scalable Oversight](https://mlsanity.com/paper/2504.18530/) (2025)
- [The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality](https://mlsanity.com/paper/2512.10791/) (2025)
- [Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labeling](https://mlsanity.com/paper/2401.08876/) (2024)
- [Explaining and Improving Information Complementarities in Multi-Agent Decision-making](https://mlsanity.com/paper/2502.06152/) (2025)
- [Select-Then-Decompose: From Empirical Analysis to Adaptive Selection Strategy for Task Decomposition in Large Language Models](https://mlsanity.com/paper/2510.17922/) (2025)
- [Factor(T,U): Factored Cognition Strengthens Monitoring of Untrusted AI](https://mlsanity.com/paper/2512.02157/) (2025)
- [Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making](https://mlsanity.com/paper/2505.18066/) (2025)
- [Human-AI Complementarity: A Goal for Amplified Oversight](https://mlsanity.com/paper/2510.26518/) (2025)
- [Recursive Decomposition with Dependencies for Generic Divide-and-Conquer Reasoning](https://mlsanity.com/paper/2505.02576/) (2025)

## Abstract (arXiv)

Human-AI complementarity, the idea that combining human and AI judgments can outperform either alone, offers a promising pathway toward robust oversight of advanced AI systems. However, whether human-AI complementarity can be achieved on realistic tasks remains an open question. We investigate this through two approaches: hybridization and two AI assistance methods (top-2 assistance and subtask delegation), evaluated on a multi-domain dataset of 1,886 samples spanning knowledge, factuality, long-context reasoning, and deception detection. We find only modest complementarity gains. Baseline hybridization yields just +0.4 percentage points (pp) over AI alone (69.3% vs 68.9%), limited both by a small complementarity region (only 8.9% of items where AI errs but humans do not) and the inability of confidence-based routing to identify it, since the model's confidence is similarly distributed across correct and incorrect predictions. Applied when AI has low confidence, top-2 assistance increases human accuracy from 28.4% to 38.3%, surpassing AI alone (37.7%) -- but primarily because humans adopt correct AI suggestions, not because they successfully override AI errors. These findings suggest that the primary bottleneck is not human task accuracy per se, but the ability to route decisions to humans when it matters and to design assistance methods that enable humans to catch AI mistakes. Our quantitative and qualitative analyses pinpoint where and why each method succeeds or fails, offering concrete targets for future work. We will release our dataset and code upon request to support progress toward more effective human-AI collaboration for AI oversight.