# Adaptive In-conversation Team Building for Language Model Agents

arXiv: https://arxiv.org/abs/2405.19425 · HTML: https://mlsanity.com/paper/2405.19425/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 62 of its 83 references that are in the map.
- First posted on arXiv 29 May 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 62)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)

## Built on by (most cited first, 30 of 39)

- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.00212/) (2025)
- [Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning](https://mlsanity.com/paper/2505.00024/) (2025)
- [Ask-before-Plan: Proactive Language Agents for Real-World Planning](https://mlsanity.com/paper/2406.12639/) (2024)
- [ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization](https://mlsanity.com/paper/2502.04306/) (2025)
- [Explaining Length Bias in LLM-Based Preference Evaluations](https://mlsanity.com/paper/2407.01085/) (2024)
- [EvoAgentX: An Automated Framework for Evolving Agentic Workflows](https://mlsanity.com/paper/2507.03616/) (2025)
- [CoAct-1: Computer-using Multi-Agent System with Coding Actions](https://mlsanity.com/paper/2508.03923/) (2025)
- [Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation](https://mlsanity.com/paper/2603.25001/) (2026)
- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch](https://mlsanity.com/paper/2511.01934/) (2025)
- [FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search](https://mlsanity.com/paper/2606.00765/) (2026)
- [EcoAct: Economic Agent Determines When to Register What Action](https://mlsanity.com/paper/2411.01643/) (2024)
- [AgentInit: Initializing LLM-based Multi-Agent Systems via Diversity and Expertise Orchestration for Effective and Efficient Collaboration](https://mlsanity.com/paper/2509.19236/) (2025)
- [Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows](https://mlsanity.com/paper/2509.23735/) (2025)
- [Tracing Agentic Failure from the Flow of Success](https://mlsanity.com/paper/2607.12747/) (2026)
- [Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives](https://mlsanity.com/paper/2502.04358/) (2025)
- [MenTeR: A fully-automated Multi-agenT workflow for end-to-end RF/Analog Circuits Netlist Design](https://mlsanity.com/paper/2505.22990/) (2025)
- [Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs](https://mlsanity.com/paper/2601.19202/) (2026)
- [Mitigating Conversational Inertia in Multi-Turn Agents](https://mlsanity.com/paper/2602.03664/) (2026)
- [Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents](https://mlsanity.com/paper/2606.26080/) (2026)
- [syftr: Pareto-Optimal Generative AI](https://mlsanity.com/paper/2505.20266/) (2025)
- [AdaptFlow: Adaptive Workflow Optimization via Meta-Learning](https://mlsanity.com/paper/2508.08053/) (2025)
- [The Path Ahead for Agentic AI: Challenges and Opportunities](https://mlsanity.com/paper/2601.02749/) (2026)
- [Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2605.29790/) (2026)
- [Memory-Augmented Agent Training for Business Document Understanding](https://mlsanity.com/paper/2412.15274/) (2024)
- [Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale](https://mlsanity.com/paper/2505.03973/) (2025)
- [Humanlike Multi-user Agent (HUMA): Designing a Deceptively Human AI Facilitator for Group Chats](https://mlsanity.com/paper/2511.17315/) (2025)
- [AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraints](https://mlsanity.com/paper/2603.13348/) (2026)
- [SPOGW: a Score-based Preference Optimization method via Group-Wise comparison for workflows](https://mlsanity.com/paper/2510.04089/) (2025)

## Abstract (arXiv)

Leveraging multiple large language model (LLM) agents has shown to be a promising approach for tackling complex tasks, while the effective design of multiple agents for a particular application remains an art. It is thus intriguing to answer a critical question: Given a task, how can we build a team of LLM agents to solve it effectively? Our new adaptive team-building paradigm offers a flexible solution, realized through a novel agent design named Captain Agent. It dynamically forms and manages teams for each step of a task-solving process, utilizing nested group conversations and reflection to ensure diverse expertise and prevent stereotypical outputs, allowing for a flexible yet structured approach to problem-solving. A comprehensive evaluation across six real-world scenarios demonstrates that Captain Agent significantly outperforms existing multi-agent methods with 21.94% improvement in average accuracy, providing outstanding performance without requiring task-specific prompt engineering. Our exploration of different backbone LLM and cost analysis further shows that Captain Agent can improve the conversation quality of weak LLM and achieve competitive performance with extremely low cost, which illuminates the application of multi-agent systems.