mlsanityData to 7 Oct 2026

AgentBench: Evaluating LLMs as Agents

arXiv 2308.03688 ↗First posted 7 Aug 2023cs.AI cs.CL cs.LGcited by 1,211

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The potential of Large Language Model (LLM) as agents has been widely acknowledged recently. Thus, there is an urgent need to quantitatively textitevaluate LLMs as agents on challenging tasks in interactive environments. We present AgentBench, a multi-dimensional benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities. Our extensive test over num API-based and open-sourced (OSS) LLMs shows that, while top commercial LLMs present a strong ability of acting as agents in complex environments, there is a significant disparity in performance between them and many OSS competitors that are no larger than 70B. We identify the typical reasons of failures in environments and LLMs, showing that poor long-term reasoning, decision-making, and instruction following abilities are the main obstacles for developing usable LLM agents. Improving instruction following and training on high quality multi-round alignment data could improve agent performance. And different from existing assumptions, training on code present ambivalent impacts on different agent tasks. Datasets, environments, and an integrated evaluation package for AgentBench are released at https://github.com/THUDM/AgentBench.

Key facts. Cited by 1,211 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 77 of its 173 references that are in the map. First posted on arXiv 7 Aug 2023.

Builds on

77 of 173 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents3 Feb
  2. 2022A Generalist Agent12 May

Other references · 75

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  13. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  14. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  15. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  16. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  17. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  18. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  19. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  20. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  21. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  22. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  23. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  24. 2023PaLM 2 Technical Reportcited by 1,154
  25. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  26. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  27. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  28. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  29. 2021Measuring Coding Challenge Competence With APPScited by 879
  30. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  31. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  32. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  33. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  34. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  35. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  36. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  37. 2022PAL: Program-aided Language Modelscited by 635
  38. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 43; 2024: 156; 2025: 272; 2026: 740.

1,211 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  7. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  8. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  9. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  10. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  11. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  12. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
38 more
  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  3. 2024Understanding the planning of LLM agents: A surveycited by 414
  4. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  5. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  6. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  7. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  8. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  9. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  10. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  11. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  12. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  13. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  14. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  15. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  16. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  17. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  18. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  19. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  20. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  21. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  22. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  23. 2024AI Agents That Mattercited by 199
  24. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  25. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  26. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  27. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  28. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  29. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  30. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  31. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  32. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  33. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 164
  34. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  35. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  36. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  37. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  38. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155

50 most cited of 1,211 All 1,211 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  2. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
  3. 2024Large Language Model Evaluation Via Multi AI Agents: Preliminary results
  4. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluation
  5. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  6. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
  7. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition
  8. 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities