mlsanityData to 14 Oct 2026

Adaptive In-conversation Team Building for Language Model Agents

arXiv 2405.19425 ↗First posted 29 May 2024cs.CLcited by 39

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Leveraging multiple large language model (LLM) agents has shown to be a promising approach for tackling complex tasks, while the effective design of multiple agents for a particular application remains an art. It is thus intriguing to answer a critical question: Given a task, how can we build a team of LLM agents to solve it effectively? Our new adaptive team-building paradigm offers a flexible solution, realized through a novel agent design named Captain Agent. It dynamically forms and manages teams for each step of a task-solving process, utilizing nested group conversations and reflection to ensure diverse expertise and prevent stereotypical outputs, allowing for a flexible yet structured approach to problem-solving. A comprehensive evaluation across six real-world scenarios demonstrates that Captain Agent significantly outperforms existing multi-agent methods with 21.94% improvement in average accuracy, providing outstanding performance without requiring task-specific prompt engineering. Our exploration of different backbone LLM and cost analysis further shows that Captain Agent can improve the conversation quality of weak LLM and achieve competitive performance with extremely low cost, which illuminates the application of multi-agent systems.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 62 of its 83 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

62 of 83 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generation15 Feb
  2. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb
  3. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration3 Oct
  4. 2023AutoAgents: A Framework for Automatic Agent Generation29 Sep
  5. 2023Agents: An Open-source Framework for Autonomous Language Agents14 Sep
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  7. 2023Cognitive Architectures for Language Agents5 Sep
  8. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  9. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug
  10. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug
  11. 2023AdaPlanner: Adaptive Planning from Feedback with Language Models26 May

Other references · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  12. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
38 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  2. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  3. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  4. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  6. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  7. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  8. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  9. 2022Large Language Models Are Reasoning Teacherscited by 329
  10. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  11. 2023Large Language Models as Tool Makerscited by 303
  12. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  13. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  14. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  15. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  16. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  17. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  18. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  19. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
  20. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197
  21. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  22. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 169
  23. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  24. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 148
  25. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 140
  26. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
  27. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 117
  28. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 107
  29. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  30. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 106
  31. 2023EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Educationcited by 80
  32. 2020Deep Learning for Source Code Modeling and Generation: Models, Applications and Challengescited by 55
  33. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  34. 2023EcoAssistant: Using LLM Assistant More Affordably and Accuratelycited by 49
  35. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  36. 2024m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Taskscited by 38
  37. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 27
  38. 2023Learning Deductive Reasoning from Synthetic Corpus based on Formal Logiccited by 18

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 18; 2026: 16.

39 papers.

Sort:

By area:

  1. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  2. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 185
  3. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 68
  4. 2024Ask-before-Plan: Proactive Language Agents for Real-World Planningcited by 60
  5. 2025ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimizationcited by 56
  6. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 49
  7. 2025EvoAgentX: An Automated Framework for Evolving Agentic Workflowscited by 44
  8. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 35
  9. 2026Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluationcited by 24
  10. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 18
  11. 2025Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratchcited by 17
  12. 2026FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Searchcited by 12
27 more
  1. 2024EcoAct: Economic Agent Determines When to Register What Actioncited by 11
  2. 2025AgentInit: Initializing LLM-based Multi-Agent Systems via Diversity and Expertise Orchestration for Effective and Efficient Collaborationcited by 10
  3. 2025Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflowscited by 10
  4. 2026Tracing Agentic Failure from the Flow of Successcited by 8
  5. 2025Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitivescited by 7
  6. 2025MenTeR: A fully-automated Multi-agenT workflow for end-to-end RF/Analog Circuits Netlist Designcited by 7
  7. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMscited by 7
  8. 2026Mitigating Conversational Inertia in Multi-Turn Agentscited by 7
  9. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agentscited by 6
  10. 2025syftr: Pareto-Optimal Generative AIcited by 5
  11. 2025AdaptFlow: Adaptive Workflow Optimization via Meta-Learningcited by 5
  12. 2026The Path Ahead for Agentic AI: Challenges and Opportunitiescited by 4
  13. 2026Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systemscited by 4
  14. 2024Memory-Augmented Agent Training for Business Document Understandingcited by 3
  15. 2025Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scalecited by 3
  16. 2025Humanlike Multi-user Agent (HUMA): Designing a Deceptively Human AI Facilitator for Group Chatscited by 2
  17. 2026AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraintscited by 2
  18. 2025SPOGW: a Score-based Preference Optimization method via Group-Wise comparison for workflowscited by 1
  19. 2026SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigationcited by 1
  20. 2025Simulating Teams with LLM Agents: Interactive 2D Environments for Studying Human-AI Dynamicscited by 0
  21. 2025The Geometry of Dialogue: Graphing Language Models to Reveal Synergistic Teams for Multi-Agent Collaborationcited by 0
  22. 2025Convergence of Outputs When Two Large Language Models Interact in a Multi-Agentic Setupcited by 0
  23. 2026Complete Cyclic Subtask Graphs for Tool-Using LLM Agents: Flexibility, Cost, and Bottlenecks in Long-Horizon Workflowscited by 0
  24. 2026From Search to Synthesis: Training LLMs as Zero-Shot Workflow Generatorscited by 0
  25. 2026ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systemscited by 0
  26. 2026Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systemscited by 0
  27. 2026BLADE: Auditing Information Use for Failure Attribution in LLM Multi-Agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving the Efficiency of Language Agent Teams with Adaptive Task Graphs
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challenges
  3. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges
  4. 2025LLM-Agent-Controller: A Universal Multi-Agent Large Language Model System as a Control Engineer
  5. 2025Literature Review Of Multi-Agent Debate For Problem-Solving
  6. 2023Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon Game
  7. 2025Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents
  8. 2024CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Models