mlsanityData to 7 Oct 2026

Survey on Evaluation of LLM-based Agents

arXiv 2503.16416 ↗First posted 20 Mar 2025cs.AI cs.CL cs.LGcited by 190

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

LLM-based agents represent a paradigm shift in AI, enabling autonomous systems to plan, reason, and use tools while interacting with dynamic environments. This paper provides the first comprehensive survey of evaluation methods for these increasingly capable agents. We analyze the field of agent evaluation across five perspectives: (1) Core LLM capabilities needed for agentic workflows, like planning, and tool use; (2) Application-specific benchmarks such as web and SWE agents; (3) Evaluation of generalist agents; (4) Analysis of agent benchmarks' core dimensions; and (5) Evaluation frameworks and tools for agent developers. Our analysis reveals current trends, including a shift toward more realistic, challenging evaluations with continuously updated benchmarks. We also identify critical gaps that future research must address, particularly in assessing cost-efficiency, safety, and robustness, and in developing fine-grained, scalable evaluation methods.

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 114 of its 172 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

114 of 172 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems19 Jan
  2. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
  3. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments4 Nov
  4. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct
  5. 2024Automated test generation to evaluate tool-augmented LLMs as conversational AI agents24 Sep
  6. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents21 Jun
  7. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  8. 2024StreamBench: Towards Benchmarking Continuous Improvement of Language Agents13 Jun
  9. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives19 Dec
  10. 2023A Survey on Large Language Model based Autonomous Agents22 Aug

Other references · 104

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
38 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2016OpenAI Gymcited by 1,684
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  13. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  14. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  15. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  16. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  17. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  18. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  19. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  20. 2021Measuring Coding Challenge Competence With APPScited by 879
  21. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  22. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  23. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  24. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  25. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  26. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  27. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  28. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  29. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  30. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  31. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  32. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  33. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  34. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  35. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  36. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  37. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  38. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348

50 most cited of 104 All 104 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 65; 2026: 125.

190 papers.

Sort:

By area:

  1. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  2. 2025Towards a Science of Scaling Agent Systemscited by 125
  3. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  4. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 120
  5. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  6. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  7. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 57
  8. 2025FutureX: An Advanced Live Benchmark for LLM Agents in Future Predictioncited by 55
  9. 2026EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoningcited by 52
  10. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 46
  11. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websitescited by 43
  12. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
38 more
  1. 2025Measuring Agents in Productioncited by 35
  2. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 32
  3. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  4. 2025ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Predictioncited by 25
  5. 2025ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filteringcited by 19
  6. 2026Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agentscited by 17
  7. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  8. 2025MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Themcited by 16
  9. 2025Risk Analysis Techniques for Governed LLM-based Multi-Agent Systemscited by 16
  10. 2025Rethinking the Reliability of Multi-agent System: A Perspective from Byzantine Fault Tolerancecited by 16
  11. 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluationcited by 15
  12. 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agentscited by 15
  13. 2025An Empirical Study of Agent Developer Practices in AI Agent Frameworkscited by 14
  14. 2025MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environmentscited by 13
  15. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  16. 2025A Practical Guide for Designing, Developing, and Deploying Production-Grade Agentic AI Workflowscited by 13
  17. 2026AI Agent Systems: Architectures, Applications, and Evaluationcited by 13
  18. 2026General Agent Evaluationcited by 13
  19. 2025BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Taskscited by 12
  20. 2025Towards Responsible and Explainable AI Agents with Consensus-Driven Reasoningcited by 12
  21. 2026SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessmentcited by 12
  22. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  23. 2025Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agentscited by 11
  24. 2025Standardization of Psychiatric Diagnoses -- Role of Fine-tuned LLM Consortium and OpenAI-gpt-oss Reasoning LLM Enabled Decision Support Systemcited by 11
  25. 2025Effective Red-Teaming of Policy-Adherent Agentscited by 10
  26. 2025MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoningcited by 10
  27. 2025Exposing LLM User Privacy via Traffic Fingerprint Analysis: A Study of Privacy Risks in LLM Agent Interactionscited by 9
  28. 2025Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlationcited by 9
  29. 2026SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewardscited by 9
  30. 2026SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agentscited by 9
  31. 2026AgentAtlas: Beyond Outcome Leaderboards for LLM Agentscited by 9
  32. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  33. 2026EnterpriseBench Corecraft: Training Generalizable Agents on High-Fidelity RL Environmentscited by 8
  34. 2026Task-Aware LLM Routing with Multi-Level Task-Profile-Guided Data Synthesis for Cold-Start Scenarioscited by 8
  35. 2026AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Trackingcited by 8
  36. 2026SEAGym: An Evaluation Environment for Self-Evolving LLM Agentscited by 8
  37. 2025Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safetycited by 7
  38. 2025PentestJudge: Judging Agent Behavior Against Operational Requirementscited by 7

50 most cited of 190 All 190 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluation and Benchmarking of LLM Agents: A Survey
  2. 2026Agent-as-a-Judge
  3. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
  4. 2024Plancraft: an evaluation dataset for planning with LLM agents
  5. 2025Benchmarking and Studying the LLM-based Agent System in End-to-End Software Development
  6. 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
  7. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  8. 2026AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities