mlsanityData to 7 Oct 2026

BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents

arXiv 2308.05960 ↗First posted 11 Aug 2023cs.AIcited by 83

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The massive successes of large language models (LLMs) encourage the emerging exploration of LLM-augmented Autonomous Agents (LAAs). An LAA is able to generate actions with its core LLM and interact with environments, which facilitates the ability to resolve complex tasks by conditioning on past interactions such as observations and actions. Since the investigation of LAA is still very recent, limited explorations are available. Therefore, we provide a comprehensive comparison of LAA in terms of both agent architectures and LLM backbones. Additionally, we propose a new strategy to orchestrate multiple LAAs such that each labor LAA focuses on one type of action, textiti.e. BOLAA, where a controller manages the communication among multiple agents. We conduct simulations on both decision-making and multi-step reasoning environments, which comprehensively justify the capacity of LAAs. Our performance results provide quantitative suggestions for designing LAA architectures and the optimal choice of LLMs, as well as the compatibility of both. We release our implementation code of LAAs to the public at urlhttps://github.com/salesforce/BOLAA.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 28 of its 34 references that are in the map. First posted on arXiv 11 Aug 2023.

Builds on

28 of 34 references.

References · 28

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
16 more
  1. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  2. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  3. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  4. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  7. 2023Language Models can Solve Computer Taskscited by 371
  8. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  9. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  10. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  11. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  12. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 110
  13. 2023Learning Performance-Improving Code Editscited by 103
  14. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66
  15. 2023DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AIcited by 21
  16. 2023REX: Rapid Exploration and eXploitation for AI Agentscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 40; 2025: 19; 2026: 6.

83 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  3. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  4. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  5. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  6. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  7. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  8. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 151
  9. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 143
  10. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  11. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  12. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
38 more
  1. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  2. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  3. 2024AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planningcited by 81
  4. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 79
  5. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  6. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 72
  7. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  8. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  9. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 68
  10. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  11. 2023RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Modelscited by 62
  12. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agentscited by 61
  13. 2023Experiential Co-Learning of Software-Developing Agentscited by 57
  14. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  15. 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Applicationcited by 54
  16. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  17. 2024AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent Systemcited by 53
  18. 2024Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agentscited by 48
  19. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  20. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Taskscited by 45
  21. 2024Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Modelscited by 45
  22. 2023ToolTalk: Evaluating Tool-Usage in a Conversational Settingcited by 42
  23. 2023LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystemcited by 35
  24. 2024Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agentscited by 35
  25. 2025MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Modelscited by 34
  26. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 30
  27. 2023Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoningcited by 30
  28. 2025Adaptive Tool Use in Large Language Models with Meta-Cognition Triggercited by 29
  29. 2025Large Language Models for Multi-Robot Systems: A Surveycited by 26
  30. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  31. 2023Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Webcited by 23
  32. 2025RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Userscited by 21
  33. 2024BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysiscited by 20
  34. 2023AI-native Interconnect Framework for Integration of Large Language Model Technologies in 6G Systemscited by 19
  35. 2024Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answeringcited by 18
  36. 2024Probabilistically Correct Language-based Multi-Robot Planning using Conformal Predictioncited by 18
  37. 2024Measuring Social Norms of Large Language Modelscited by 16
  38. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges
  2. 2025Graph-Augmented Large Language Model Agents: Current Progress and Future Prospects
  3. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  4. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challenges
  5. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
  6. 2024Practical Considerations for Agentic LLM Systems
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Survey
  8. 2026The Path Ahead for Agentic AI: Challenges and Opportunities