mlsanityData to 7 Oct 2026

AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

arXiv 2401.13178 ↗First posted 24 Jan 2024cs.CL cs.AI cs.LGcited by 319

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Evaluating Large Language Models (LLMs) as general-purpose agents is essential for understanding their capabilities and facilitating their integration into practical applications. However, the evaluation process presents substantial challenges. A primary obstacle is the benchmarking of agent performance across diverse scenarios within a unified framework, especially in maintaining partially-observable environments and ensuring multi-round interactions. Moreover, current evaluation frameworks mostly focus on the final success rate, revealing few insights during the process and failing to provide a deep understanding of the model abilities. To address these challenges, we introduce AgentBoard, a pioneering comprehensive benchmark and accompanied open-source evaluation framework tailored to analytical evaluation of LLM agents. AgentBoard offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit that features easy assessment of agents for multi-faceted analysis. This not only sheds light on the capabilities and limitations of LLM agents but also propels the interpretability of their performance to the forefront. Ultimately, AgentBoard serves as a step towards demystifying agent behaviors and accelerating the development of stronger LLM agents.

Key facts. Cited by 319 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 52 of its 65 references that are in the map. First posted on arXiv 24 Jan 2024.

Builds on

52 of 65 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Evaluating Language-Model Agents on Realistic Autonomous Tasks18 Dec
  2. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents20 Nov
  3. 2023OpenAgents: An Open Platform for Language Agents in the Wild16 Oct
  4. 2023Lemur: Harmonizing Natural Language and Code for Language Agents10 Oct
  5. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents11 Aug
  6. 2023Gentopia: A Collaborative Platform for Tool-Augmented LLMs8 Aug
  7. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug
  9. 2022A Generalist Agent12 May

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023Mistral 7Bcited by 3,065
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
31 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  12. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  13. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  14. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  15. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  16. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  17. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  18. 2023Augmented Language Models: a Surveycited by 398
  19. 2023Tool Learning with Foundation Modelscited by 302
  20. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  21. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  22. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  23. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  24. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  25. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  26. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  27. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  28. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  29. 2018CEM-RL: Combining evolutionary and gradient-based methods for policy searchcited by 53
  30. 2020PDDLGym: Gym Environments from PDDL Problemscited by 45
  31. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 38; 2025: 85; 2026: 196.

319 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  3. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  4. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  5. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  6. 2024AI Agents That Mattercited by 199
  7. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  9. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  10. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  11. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  12. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Spacecited by 141
38 more
  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  2. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  3. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  4. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  5. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  6. 2025Agentic Large Language Models, a surveycited by 105
  7. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  8. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  9. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  10. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  11. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 73
  12. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agentscited by 72
  13. 2024OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automationcited by 66
  14. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  15. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
  16. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 56
  17. 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentscited by 56
  18. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  19. 2025AgentRefine: Enhancing Agent Generalization through Refinement Tuningcited by 51
  20. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  21. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  22. 2025EvoAgentX: An Automated Framework for Evolving Agentic Workflowscited by 43
  23. 2024On the Multi-turn Instruction Following for Conversational Web Agentscited by 42
  24. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 40
  25. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agentscited by 39
  26. 2024ACPBench: Reasoning about Action, Change, and Planningcited by 37
  27. 2025STeCa: Step-level Trajectory Calibration for LLM Agent Learningcited by 37
  28. 2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
  29. 2025MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Modelscited by 34
  30. 2025Structured Uncertainty guided Clarification for LLM Agentscited by 34
  31. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  32. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agentscited by 31
  33. 2025ReflAct: World-Grounded Decision Making in LLM Agents via Goal-State Reflectioncited by 30
  34. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  35. 2025ATLaS: Agent Tuning via Learning Critical Stepscited by 29
  36. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Modelscited by 28
  37. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 28
  38. 2024ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Modelscited by 25

50 most cited of 319 All 319 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
  2. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  3. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
  4. 2025LLM-Agent-Controller: A Universal Multi-Agent Large Language Model System as a Control Engineer
  5. 2024Large Language Model Evaluation Via Multi AI Agents: Preliminary results
  6. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluation
  7. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  8. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments