mlsanityData to 14 Oct 2026

AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents

arXiv 2404.06411 ↗First posted 9 Apr 2024cs.AI cs.CLcited by 28

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The advances made by Large Language Models (LLMs) have led to the pursuit of LLM agents that can solve intricate, multi-step reasoning tasks. As with any research pursuit, benchmarking and evaluation are key corner stones to efficient and reliable progress. However, existing benchmarks are often narrow and simply compute overall task success. To face these issues, we propose AgentQuest -- a framework where (i) both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs; (ii) we offer two new evaluation metrics that can reliably track LLM agent progress while solving a task. We exemplify the utility of the metrics on two use cases wherein we identify common failure points and refine the agent architecture to obtain a significant performance increase. Together with the research community, we hope to extend AgentQuest further and therefore we make it available under https://github.com/nec-research/agentquest.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 8 of its 21 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

8 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Survey on Large Language Model based Autonomous Agents22 Aug

Other references · 7

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  4. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  5. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  6. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  7. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 54

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 9; 2026: 13.

28 papers.

Sort:

By area:

  1. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  2. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  3. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  4. 2024Advancing Agentic Systems: Dynamic Task Decomposition, Tool Integration and Evaluation using Novel Metrics and Datasetcited by 24
  5. 2025Pre-Act: Multi-Step Planning and Reasoning Improves Acting in LLM Agentscited by 24
  6. 2025MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosiscited by 23
  7. 2024AutoPenBench: Benchmarking Generative Agents for Penetration Testingcited by 21
  8. 2026Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systemscited by 20
  9. 2024Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applicationscited by 19
  10. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 11
  11. 2025STEER-ME: Assessing the Microeconomic Reasoning of Large Language Modelscited by 7
  12. 2025MAPS: A Multilingual Benchmark for Agent Performance and Securitycited by 7
16 more
  1. 2026TIDE: Trajectory-based Diagnostic Evaluation of Test-Time Improvement in LLM Agentscited by 7
  2. 2026PrefIx: Understand and Adapt to User Preference in Human-Agent Interactioncited by 7
  3. 2025Are Autonomous Web Agents Good Testers?cited by 6
  4. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 6
  5. 2026Generative AI collective behavior needs an interactionist paradigmcited by 6
  6. 2026CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization Vcited by 2
  7. 2026Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safetycited by 2
  8. 2026Redundant or Necessary? A Benchmark for Detecting Redundant Steps in Agent Trajectoriescited by 2
  9. 2025Plant in Cupboard, Orange on Rably, Inat Aphone. Benchmarking Incremental Learning of Situation and Language Model using a Text-Simulated Situated Environmentcited by 1
  10. 2026Beyond End-to-End Success: Diagnosing Failures in Long-Horizon Security LLM Agentscited by 1
  11. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  12. 2026Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discoverycited by 1
  13. 2025COCORELI: Enforcing Execution Preconditions for Reliable Collaborative Instruction Followingcited by 0
  14. 2026Latent Action Reparameterization for Efficient Agent Inferencecited by 0
  15. 2026Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AIcited by 0
  16. 2026AgentStripes: An Evidence-Graded Maturity Model for AI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
  2. 2023AgentBench: Evaluating LLMs as Agents
  3. 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
  4. 2026AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
  5. 2024ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents
  6. 2026Benchmark Everything Everywhere All at Once
  7. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Space
  8. 2026EvalBuilder: Can LLMs Build Efficient Benchmarks?