# TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use

arXiv: https://arxiv.org/abs/2510.04550 · HTML: https://mlsanity.com/paper/2510.04550/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 45 of its 56 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 30 of 45)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)

## Built on by (most cited first, 30 of 51)

- [ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents](https://mlsanity.com/paper/2601.12294/) (2026)
- [The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration](https://mlsanity.com/paper/2603.22862/) (2026)
- [EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL](https://mlsanity.com/paper/2605.18703/) (2026)
- [FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search](https://mlsanity.com/paper/2606.00765/) (2026)
- [PhGPO: Pheromone-Guided Policy Optimization for Long-Horizon Tool Planning](https://mlsanity.com/paper/2602.13691/) (2026)
- [PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents](https://mlsanity.com/paper/2608.04003/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [ToolPrivacyBench: Benchmarking Purpose-Bound Privacy in Tool-Using LLM Agents](https://mlsanity.com/paper/2606.28061/) (2026)
- [Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?](https://mlsanity.com/paper/2606.29920/) (2026)
- [SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior](https://mlsanity.com/paper/2606.11543/) (2026)
- [GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation](https://mlsanity.com/paper/2606.22737/) (2026)
- [What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents](https://mlsanity.com/paper/2607.06184/) (2026)
- [ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration](https://mlsanity.com/paper/2601.06860/) (2026)
- [ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox](https://mlsanity.com/paper/2605.10787/) (2026)
- [Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development](https://mlsanity.com/paper/2608.13417/) (2026)
- [AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems](https://mlsanity.com/paper/2601.09393/) (2026)
- [UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents](https://mlsanity.com/paper/2604.11557/) (2026)
- [Long-Horizon Plan Execution in Large Tool Spaces through Entropy-Guided Branching](https://mlsanity.com/paper/2604.12126/) (2026)
- [TimeClaw: A Time-Series AI Agent with Exploratory Execution Learning](https://mlsanity.com/paper/2605.10038/) (2026)
- [Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values](https://mlsanity.com/paper/2605.10365/) (2026)
- [PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections](https://mlsanity.com/paper/2606.12737/) (2026)
- [trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories](https://mlsanity.com/paper/2609.00038/) (2026)
- [The Menu Is an Execution Prior: State-Path Tool Menus for Online Agents](https://mlsanity.com/paper/2609.09395/) (2026)
- [Persuasion Propagation: Does Persuasion Change AI Agent Behavior?](https://mlsanity.com/paper/2602.00851/) (2026)
- [When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors](https://mlsanity.com/paper/2604.21255/) (2026)
- [Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows](https://mlsanity.com/paper/2604.25345/) (2026)
- [Hybrid Inspection and Task-Based Access Control in Zero-Trust Agentic AI](https://mlsanity.com/paper/2605.02682/) (2026)
- [Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows](https://mlsanity.com/paper/2605.24219/) (2026)
- [Binding Drift in Multi-Step Tool-Augmented Agents](https://mlsanity.com/paper/2607.18316/) (2026)
- [DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making](https://mlsanity.com/paper/2607.20491/) (2026)

## Abstract (arXiv)

Large language model (LLM)-based agents increasingly rely on tool use to complete real-world tasks. While existing works evaluate the LLMs' tool use capability, they largely focus on the final answers yet overlook the detailed tool usage trajectory, i.e., whether tools are selected, parameterized, and ordered correctly. We introduce TRAJECT-Bench, a trajectory-aware benchmark to comprehensively evaluate LLMs' tool use capability through diverse tasks with fine-grained evaluation metrics. TRAJECT-Bench pairs high-fidelity, executable tools across practical domains with tasks grounded in production-style APIs, and synthesizes trajectories that vary in breadth (parallel calls) and depth (interdependent chains). Besides final accuracy, TRAJECT-Bench also reports trajectory-level diagnostics, including tool selection and argument correctness, and dependency/order satisfaction. Analyses reveal failure modes such as similar tool confusion and parameter-blind selection, and scaling behavior with tool diversity and trajectory length where the bottleneck of transiting from short to mid-length trajectories is revealed, offering actionable guidance for LLMs' tool use.