# Designing Algorithmic Delegates: The Role of Indistinguishability in Human-AI Handoff

arXiv: https://arxiv.org/abs/2506.03102 · HTML: https://mlsanity.com/paper/2506.03102/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 55 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 15 of 15)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Towards Human-AI Complementarity with Prediction Sets](https://mlsanity.com/paper/2405.17544/) (2024)
- [Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function](https://mlsanity.com/paper/2406.01382/) (2024)
- [Designing Skill-Compatible AI: Methodologies and Frameworks in Chess](https://mlsanity.com/paper/2405.05066/) (2024)
- [Taking Advice from (Dis)Similar Machines: The Impact of Human-Machine Similarity on Machine-Assisted Decision-Making](https://mlsanity.com/paper/2209.03821/) (2022)
- [Learning to Defer in Congested Systems: The AI-Human Interplay](https://mlsanity.com/paper/2402.12237/) (2024)
- [Novel Dense Subgraph Discovery Primitives: Risk Aversion and Exclusion Queries](https://mlsanity.com/paper/1904.08178/) (2019)
- [Persuasion, Delegation, and Private Information in Algorithm-Assisted Decisions](https://mlsanity.com/paper/2402.09384/) (2024)
- [The Value of Context: Human versus Black Box Evaluators](https://mlsanity.com/paper/2402.11157/) (2024)

## Built on by (most cited first, 5 of 5)

- [What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commerce](https://mlsanity.com/paper/2508.02630/) (2025)
- [Collaborating with GenAI: Incentives and Replacements](https://mlsanity.com/paper/2508.20213/) (2025)
- [Algorithmic Feature Highlighting for Human-AI Decision-Making](https://mlsanity.com/paper/2604.22236/) (2026)
- [Human Decision-Making with AI Assistance under Correlated Features](https://mlsanity.com/paper/2606.20628/) (2026)
- [Provably Optimal Learning Algorithms for Assistance Games](https://mlsanity.com/paper/2607.08012/) (2026)

## Abstract (arXiv)

As AI technologies improve, people are increasingly willing to delegate tasks to AI agents. In many cases, the human decision-maker chooses whether to delegate to an AI agent based on properties of the specific instance of the decision-making problem they are facing. Since humans typically lack full awareness of all the factors relevant to this choice for a given decision-making instance, they perform a kind of categorization by treating indistinguishable instances -- those that have the same observable features -- as the same. In this paper, we define the problem of designing the optimal algorithmic delegate in the presence of categories. This is an important dimension in the design of algorithms to work with humans, since we show that the optimal delegate can be an arbitrarily better teammate than the optimal standalone algorithmic agent. The solution to this optimal delegation problem is not obvious: we discover that this problem is fundamentally combinatorial, and illustrate the complex relationship between the optimal design and the properties of the decision-making task even in simple settings. Indeed, we show that finding the optimal delegate is computationally hard in general. However, we are able to find efficient algorithms for producing the optimal delegate in several broad cases of the problem, including when the optimal action may be decomposed into functions of features observed by the human and the algorithm. Finally, we run computational experiments to simulate a designer updating an algorithmic delegate over time to be optimized for when it is actually adopted by users, and show that while this process does not recover the optimal delegate in general, the resulting delegate often performs quite well.