# When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-making arXiv: https://arxiv.org/abs/2308.11721 · HTML: https://mlsanity.com/paper/2308.11721/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 56 references that are in the map. - First posted on arXiv 22 Aug 2023. ## Builds on (most cited first, 30 of 34) - [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020) - [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020) - [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017) - [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022) - [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019) - [Algorithmic Monoculture and Social Welfare](https://mlsanity.com/paper/2101.05853/) (2021) - [A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores](https://mlsanity.com/paper/2002.08035/) (2020) - [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023) - [Direct Uncertainty Prediction for Medical Second Opinions](https://mlsanity.com/paper/1807.01771/) (2018) - [Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness](https://mlsanity.com/paper/2202.08821/) (2022) - [Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts](https://mlsanity.com/paper/2206.07948/) (2022) - [Human-AI Collaboration with Bandit Feedback](https://mlsanity.com/paper/2105.10614/) (2021) - [On the Utility of Prediction Sets in Human-AI Teams](https://mlsanity.com/paper/2205.01411/) (2022) - [Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging](https://mlsanity.com/paper/2205.09696/) (2022) - [Human-Aligned Calibration for AI-Assisted Decision Making](https://mlsanity.com/paper/2306.00074/) (2023) - [How Bad is Top-K Recommendation under Competing Content Creators?](https://mlsanity.com/paper/2302.01971/) (2023) - [Designing Decision Support Systems Using Counterfactual Prediction Sets](https://mlsanity.com/paper/2306.03928/) (2023) - [When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming](https://mlsanity.com/paper/2306.04930/) (2023) - [Incentivizing Exploration with Selective Data Disclosure](https://mlsanity.com/paper/1811.06026/) (2018) - [Individual Fairness in Pipelines](https://mlsanity.com/paper/2004.05167/) (2020) - [The Assistive Multi-Armed Bandit](https://mlsanity.com/paper/1901.08654/) (2019) - [Improving Screening Processes via Calibrated Subset Selection](https://mlsanity.com/paper/2202.01147/) (2022) - [On component interactions in two-stage recommender systems](https://mlsanity.com/paper/2106.14979/) (2021) - [Algorithmic Assistance with Recommendation-Dependent Preferences](https://mlsanity.com/paper/2208.07626/) (2022) - [Auditing for Human Expertise](https://mlsanity.com/paper/2306.01646/) (2023) - [Towards Modeling and Influencing the Dynamics of Human Learning](https://mlsanity.com/paper/2301.00901/) (2023) - [Fairness Incentives for Myopic Agents](https://mlsanity.com/paper/1705.02321/) (2017) - [A Bandit Model for Human-Machine Decision Making with Private Information and Opacity](https://mlsanity.com/paper/2007.04800/) (2020) - [Uncertainty Quantification for Fairness in Two-Stage Recommender Systems](https://mlsanity.com/paper/2205.15436/) (2022) - [Incentivizing Combinatorial Bandit Exploration](https://mlsanity.com/paper/2206.00494/) (2022) ## Built on by (most cited first, 4 of 4) - [Towards Human-AI Complementarity with Prediction Sets](https://mlsanity.com/paper/2405.17544/) (2024) - [No Need for Learning to Defer? A Training Free Deferral Framework to Multiple Experts through Conformal Prediction](https://mlsanity.com/paper/2509.12573/) (2025) - [Strategic Decision Support for AI Agents](https://mlsanity.com/paper/2606.12587/) (2026) - [DeCoDe: Defer-and-Complement Decision-Making via Decoupled Concept Bottleneck Models](https://mlsanity.com/paper/2505.19220/) (2025) ## Abstract (arXiv) Historically, much of machine learning research has focused on the performance of the algorithm alone, but recently more attention has been focused on optimizing joint human-algorithm performance. Here, we analyze a specific type of human-algorithm collaboration where the algorithm has access to a set of n items, and presents a subset of size k to the human, who selects a final item from among those k. This scenario could model content recommendation, route planning, or any type of labeling task. Because both the human and algorithm have imperfect, noisy information about the true ordering of items, the key question is: which value of k maximizes the probability that the best item will be ultimately selected? For k=1, performance is optimized by the algorithm acting alone, and for k=n it is optimized by the human acting alone. Surprisingly, we show that for multiple of noise models, it is optimal to set k ∈ [2, n-1] - that is, there are strict benefits to collaborating, even when the human and algorithm have equal accuracy separately. We demonstrate this theoretically for the Mallows model and experimentally for the Random Utilities models of noisy permutations. However, we show this pattern is reversed when the human is anchored on the algorithm's presented ordering - the joint system always has strictly worse performance. We extend these results to the case where the human and algorithm differ in their accuracy levels, showing that there always exist regimes where a more accurate agent would strictly benefit from collaborating with a less accurate one, but these regimes are asymmetric between the human and the algorithm's accuracy.