mlsanityData to 7 Oct 2026

The Amazing Agent Race: Strong Tool Users, Weak Navigators

arXiv 2604.10261 ↗First posted 11 Apr 2026cs.AI cs.CL cs.LGcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Existing tool-use benchmarks for LLM agents are overwhelmingly linear: our analysis of six benchmarks shows 55 to 100% of instances are simple chains of 2 to 5 steps. We introduce The Amazing Agent Race (AAR), a benchmark featuring directed acyclic graph (DAG) puzzles (or "legs") with fork-merge tool chains. We release 1,400 instances across two variants: sequential (800 legs) and compositional (600 DAG legs). Agents must navigate Wikipedia, execute multi-step tool chains, and aggregate results into a verifiable answer. Legs are procedurally generated from Wikipedia seeds across four difficulty levels with live-API validation. Three complementary metrics (finish-line accuracy, pit-stop visit rate, and roadblock completion rate) separately diagnose navigation, tool-use, and arithmetic failures. Evaluating three agent frameworks on 1,400 legs, the best achieves only 37.2% accuracy. Navigation errors dominate (27 to 52% of trials) while tool-use errors remain below 17%, and agent architecture matters as much as model scale (Claude Code matches Codex CLI at 37% with 6x fewer tokens). The compositional structure of AAR reveals that agents fail not at calling tools but at navigating to the right pages, a blind spot invisible to linear benchmarks. The project page can be accessed at: https://minnesotanlp.github.io/the-amazing-agent-race

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 20 of its 25 references that are in the map. First posted on arXiv 11 Apr 2026.

Builds on

20 of 25 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  3. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents24 Jan
  4. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2018Datasheets for Datasetscited by 1,371
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  7. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  9. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  10. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  11. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  12. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138
4 more
  1. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
  2. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 91
  3. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 48
  4. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Callscited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentEscapeBench: Evaluating Out-of-Domain Tool-Grounded Reasoning in LLM Agents
  2. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  3. 2026AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
  4. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
  5. 2026Benchmark Test-Time Scaling of General LLM Agents
  6. 2026E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
  7. 2026AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
  8. 2026ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?