# When Should We Orchestrate Multiple Agents?

arXiv: https://arxiv.org/abs/2503.13577 · HTML: https://mlsanity.com/paper/2503.13577/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 17 Mar 2025.

## Builds on (most cited first, 25 of 25)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [When combinations of humans and AI are useful: A systematic review and meta-analysis](https://mlsanity.com/paper/2405.06087/) (2024)
- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [Towards Unbiased and Accurate Deferral to Multiple Experts](https://mlsanity.com/paper/2102.13004/) (2021)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Sample Efficient Learning of Predictors that Complement Humans](https://mlsanity.com/paper/2207.09584/) (2022)
- [Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts](https://mlsanity.com/paper/2206.07948/) (2022)
- [Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegability](https://mlsanity.com/paper/1902.03245/) (2019)
- [Human-AI Collaboration with Bandit Feedback](https://mlsanity.com/paper/2105.10614/) (2021)
- [Teaching Humans When To Defer to a Classifier via Exemplars](https://mlsanity.com/paper/2111.11297/) (2021)
- [Online Learning with Abstention](https://mlsanity.com/paper/1703.03478/) (2017)
- [A Decision Theoretic Framework for Measuring AI Reliance](https://mlsanity.com/paper/2401.15356/) (2024)
- [Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressure](https://mlsanity.com/paper/2306.07458/) (2023)
- [Learning Personalized Decision Support Policies](https://mlsanity.com/paper/2304.06701/) (2023)
- [Modulating Language Model Experiences through Frictions](https://mlsanity.com/paper/2407.12804/) (2024)
- [Optimizing delegation between human and AI collaborative agents](https://mlsanity.com/paper/2309.14718/) (2023)
- [Learning Complementary Policies for Human-AI Teams](https://mlsanity.com/paper/2302.02944/) (2023)
- [Representational Alignment Supports Effective Machine Teaching](https://mlsanity.com/paper/2406.04302/) (2024)
- [Revisiting Rogers' Paradox in the Context of Human-AI Interaction](https://mlsanity.com/paper/2501.10476/) (2025)

## Built on by (most cited first, 7 of 7)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspective](https://mlsanity.com/paper/2504.03255/) (2025)
- [Terrarium: Revisiting the Blackboard for Multi-Agent Safety, Privacy, and Security Studies](https://mlsanity.com/paper/2510.14312/) (2025)
- [UFO3: Weaving the Digital Agent Galaxy](https://mlsanity.com/paper/2511.11332/) (2025)
- [Why Human Guidance Matters in Collaborative Vibe Coding](https://mlsanity.com/paper/2602.10473/) (2026)
- [Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts](https://mlsanity.com/paper/2510.07205/) (2025)
- [Multi-Agent Coordination Adaptation via Structure-Guided Orchestration](https://mlsanity.com/paper/2605.25746/) (2026)

## Abstract (arXiv)

Strategies for orchestrating the interactions between multiple agents, both human and artificial, can wildly overestimate performance and underestimate the cost of orchestration. We design a framework to orchestrate agents under realistic conditions, such as inference costs or availability constraints. We show theoretically that orchestration is only effective if there are performance or cost differentials between agents. We then empirically demonstrate how orchestration between multiple agents can be helpful for selecting agents in a simulated environment, picking a learning strategy in the infamous Rogers' Paradox from social science, and outsourcing tasks to other agents during a question-answer task in a user study.