mlsanityData to 7 Oct 2026

AI Agents That Matter

arXiv 2407.01502 ↗First posted 1 Jul 2024cs.LG cs.AIcited by 199

Not on the map. Why?

Cited by 123 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

AI agents are an exciting new research direction, and agent development is driven by benchmarks. Our analysis of current agent benchmarks and evaluation practices reveals several shortcomings that hinder their usefulness in real-world applications. First, there is a narrow focus on accuracy without attention to other metrics. As a result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains. Our focus on cost in addition to accuracy motivates the new goal of jointly optimizing the two metrics. We design and implement one such optimization, showing its potential to greatly reduce cost while maintaining accuracy. Second, the benchmarking needs of model and downstream developers have been conflated, making it hard to identify which agent would be best suited for a particular application. Third, many agent benchmarks have inadequate holdout sets, and sometimes none at all. This has led to agents that are fragile because they take shortcuts and overfit to the benchmark in various ways. We prescribe a principled framework for avoiding overfitting. Finally, there is a lack of standardization in evaluation practices, leading to a pervasive lack of reproducibility. We hope that the steps we introduce for addressing these shortcomings will spur the development of agents that are useful in the real world and not just accurate on benchmarks.

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). Builds on 50 of its 98 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

50 of 98 references.

References · 50

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
38 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  5. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  6. 2017Deep Reinforcement Learning that Matterscited by 727
  7. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  8. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  9. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  10. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  11. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  12. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  13. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  14. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  15. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  16. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  17. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  18. 2024More Agents Is All You Needcited by 174
  19. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  20. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  21. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138
  22. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  23. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  24. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 119
  25. 2018Troubling Trends in Machine Learning Scholarshipcited by 107
  26. 2024The Ethics of Advanced AI Assistantscited by 103
  27. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  28. 2024Can Language Models Solve Olympiad Programming?cited by 70
  29. 2023SteP: Stacked LLM Policies for Web Actionscited by 66
  30. 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
  31. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agentscited by 63
  32. 2023Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoningcited by 49
  33. 2024WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Settingcited by 47
  34. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41
  35. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agentscited by 39
  36. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  37. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  38. 2024On the Brittle Foundations of ReAct Prompting for Agentic Large Language Modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 61; 2026: 123.

199 papers.

Sort:

By area:

  1. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  2. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  3. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  4. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 150
  5. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  6. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  7. 2025Towards a Science of Scaling Agent Systemscited by 125
  8. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
  9. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  10. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  11. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 79
  12. 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directionscited by 74
38 more
  1. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  2. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  3. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  4. 2025Levels of Autonomy for AI Agentscited by 45
  5. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  6. 2025Infrastructure for AI Agentscited by 40
  7. 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 38
  8. 2025Cost-of-Pass: An Economic Framework for Evaluating Language Modelscited by 35
  9. 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systemscited by 33
  10. 2024What Makes Large Language Models Reason in (Multi-Turn) Code Generation?cited by 29
  11. 2026APEX-Agentscited by 29
  12. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  13. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  14. 2025Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoningcited by 27
  15. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 27
  16. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  17. 2026MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarkscited by 26
  18. 2024Provable Scaling Laws for the Test-Time Compute of Large Language Modelscited by 25
  19. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 25
  20. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  21. 2024Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applicationscited by 19
  22. 2025FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agentscited by 16
  23. 2025Evaluating the Goal-Directedness of Large Language Modelscited by 15
  24. 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptionscited by 15
  25. 2025The AI Agent Indexcited by 14
  26. 2025OvercookedV2: Rethinking Overcooked for Zero-Shot Coordinationcited by 14
  27. 2025An Empirical Study of Agent Developer Practices in AI Agent Frameworkscited by 14
  28. 2026General Agent Evaluationcited by 13
  29. 2025Integrating Large Language Models for Automated Structural Analysiscited by 12
  30. 2026Autonomous Agents on Blockchains: Standards, Execution Models, and Trust Boundariescited by 12
  31. 2026Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboardscited by 12
  32. 2025Measuring AI agent autonomy: Towards a scalable approach with code inspectioncited by 10
  33. 2025Automated Composition of Agents: A Knapsack Approach for Agentic Component Selectioncited by 10
  34. 2024TourSynbio: A Multi-Modal Large Model and Agent Framework to Bridge Text and Protein Sequences for Protein Engineeringcited by 9
  35. 2025AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Securitycited by 9
  36. 2026SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agentscited by 9
  37. 2026AgentAtlas: Beyond Outcome Leaderboards for LLM Agentscited by 9
  38. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9

50 most cited of 199 All 199 (JSON)