mlsanityData to 14 Oct 2026

Establishing Best Practices for Building Rigorous Agentic Benchmarks

arXiv 2507.02825 ↗First posted 3 Jul 2025cs.AIcited by 102

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities by evaluating task outcomes via specific reward designs. However, we show that many agentic benchmarks have issues in task setup or reward design. For example, SWE-bench Verified uses insufficient test cases, while TAU-bench counts empty responses as successful. Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms. To make agentic evaluation rigorous, we introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-Bench, a benchmark with a particularly complex evaluation design, ABC reduces the performance overestimation by 33%.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 51 of its 95 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

51 of 95 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  3. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment19 Feb
  4. 2023Evaluating Language-Model Agents on Realistic Autonomous Tasks18 Dec
  5. 2023AutoAgents: A Framework for Automatic Agent Generation29 Sep
  6. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks27 May

Other references · 45

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  10. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  12. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
33 more
  1. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  2. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  3. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  4. 2024Measuring short-form factuality in large language modelscited by 373
  5. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  6. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  7. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  8. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  9. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  10. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 225
  11. 2024AI Agents That Mattercited by 219
  12. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  13. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  14. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 164
  15. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 115
  16. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 114
  17. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  18. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  19. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 96
  20. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  21. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  22. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
  23. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  24. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  25. 2024The Limits of Inference Scaling Through Resamplingcited by 53
  26. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 51
  27. 2025UTBoost: Rigorous Evaluation of Coding Agents on SWE-Benchcited by 45
  28. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  29. 2024Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmarkcited by 34
  30. 2023Evaluating Cross-Domain Text-to-SQL Models and Benchmarkscited by 32
  31. 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 22
  32. 2024More than Marketing? On the Information Value of AI Benchmarks for Practitionerscited by 17
  33. 2024Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budgetcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 94.

102 papers.

Sort:

By area:

  1. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 315
  2. 2025Towards a Science of Scaling Agent Systemscited by 138
  3. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 54
  4. 2026Quantifying Frontier LLM Capabilities for Container Sandbox Escapecited by 18
  5. 2026Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learnedcited by 18
  6. 2026AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluationcited by 15
  7. 2026Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboardscited by 13
  8. 2026KernelFoundry: Hardware-aware evolutionary GPU kernel optimizationcited by 13
  9. 2026Failure as a Process: An Anatomy of CLI Coding Agent Trajectoriescited by 13
  10. 2025OpenApps: Simulating Environment Variations to Measure UI-Agent Reliabilitycited by 12
  11. 2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 12
  12. 2025ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interactioncited by 11
38 more
  1. 2025STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reportscited by 10
  2. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  3. 2026Efficient Benchmarking of AI Agentscited by 10
  4. 2026SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agentscited by 10
  5. 2026TeamBench: Evaluating Agent Coordination under Enforced Role Separationcited by 10
  6. 2026SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarioscited by 10
  7. 2026Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failurescited by 10
  8. 2026Measuring AI R&D Automationcited by 9
  9. 2026The Illusion of Multi-Agent Advantagecited by 9
  10. 2026KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?cited by 8
  11. 2026From Governance Norms to Enforceable Controls: A Layered Translation Method for Runtime Guardrails in Agentic AIcited by 7
  12. 2026Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repaircited by 6
  13. 2026Seven simple steps for log analysis in AI systemscited by 6
  14. 2026ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agentscited by 6
  15. 2026Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agentscited by 6
  16. 2026Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wildcited by 5
  17. 2026SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cyclecited by 5
  18. 2026Anchor: Mitigating Artifact Drift in Agent Benchmark Generationcited by 5
  19. 2026Life After Benchmark Saturation: A Case Study of CORE-Benchcited by 4
  20. 2026Building to the Test: Coding Agents Deliver What You Check, Not What You Requestedcited by 4
  21. 2026HarnessEval-W: Agentifying the Evaluation of Visual Worldscited by 4
  22. 2026On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecificationcited by 4
  23. 2026AgentOS: From Application Silos to a Natural Language-Driven Data Ecosystemcited by 3
  24. 2026Autonomy and Agency in Agentic AI: Architectural Tactics for Regulated Contextscited by 3
  25. 2026Interactive Evaluation Requires a Design Sciencecited by 3
  26. 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safetycited by 3
  27. 2026AgentFairBench: Do LLM Agents Discriminate When They Act?cited by 3
  28. 2026Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Taskscited by 3
  29. 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineeringcited by 3
  30. 2026Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agentscited by 3
  31. 2025Mapping Overlaps in Benchmarks through Perplexity in the Wildcited by 2
  32. 2025UpBench: A Dynamically Evolving Real-World Labor-Market Agentic Benchmark Framework Built for Human-Centric AIcited by 2
  33. 2026Beyond Rule-Based Workflows: An Information-Flow-Orchestrated Multi-Agents Paradigm via Agent-to-Agent Communication from CORALcited by 2
  34. 2026Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AIcited by 2
  35. 2026SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluationscited by 2
  36. 2026Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational studycited by 2
  37. 2026Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoningcited by 2
  38. 2026OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agentscited by 2

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
  2. 2025Beyond Black-Box Benchmarking: Observability, Analytics, and Optimization of Agentic Systems
  3. 2025Towards a Standard, Enterprise-Relevant Agentic AI Benchmark: Lessons from 5.5 billion tokens' worth of agentic AI evaluations
  4. 2026The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean
  5. 2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
  6. 2026Benchmark Everything Everywhere All at Once
  7. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
  8. 2026AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints