# Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments arXiv: https://arxiv.org/abs/2609.37315 · HTML: https://mlsanity.com/paper/2609.37315/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 49 references that are in the map. - First posted on arXiv 29 Sep 2026. ## Builds on (most cited first, 30 of 30) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024) - [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025) - [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024) - [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024) - [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024) - [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025) - [BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices](https://mlsanity.com/paper/2411.12990/) (2024) - [Establishing Best Practices for Building Rigorous Agentic Benchmarks](https://mlsanity.com/paper/2507.02825/) (2025) - [UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench](https://mlsanity.com/paper/2506.09289/) (2025) - [Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agents](https://mlsanity.com/paper/2602.22302/) (2026) - [ToolFuzz -- Automated Agent Tool Testing](https://mlsanity.com/paper/2503.04479/) (2025) - [Defining and Detecting the Defects of the Large Language Model-based Autonomous Agents](https://mlsanity.com/paper/2412.18371/) (2024) - [ToolGate: Contract-Grounded and Verified Tool Execution for LLMs](https://mlsanity.com/paper/2601.04688/) (2026) - [BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks](https://mlsanity.com/paper/2604.24955/) (2026) - [From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents](https://mlsanity.com/paper/2606.09863/) (2026) - [Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation](https://mlsanity.com/paper/2602.11224/) (2026) - [Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes](https://mlsanity.com/paper/2603.06847/) (2026) - [Automated Benchmark Auditing for AI Agents and Large Language Models](https://mlsanity.com/paper/2605.26079/) (2026) - [LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks](https://mlsanity.com/paper/2604.13072/) (2026) - [Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents](https://mlsanity.com/paper/2604.16706/) (2026) - [JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems](https://mlsanity.com/paper/2604.23478/) (2026) - [Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation](https://mlsanity.com/paper/2607.02577/) (2026) - [Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation](https://mlsanity.com/paper/2605.10448/) (2026) - [Contract2Tool: Learning Preconditions and Effects for Reliable Tool-Augmented LLM Agents](https://mlsanity.com/paper/2606.07904/) (2026) - [MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents](https://mlsanity.com/paper/2607.11818/) (2026) - [Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety](https://mlsanity.com/paper/2603.18245/) (2026) - [ContractBench: Can LLM Agents Preserve Observation Contracts?](https://mlsanity.com/paper/2605.17281/) (2026) - [Systematic API Testing Through Model Checking and Executable Contracts](https://mlsanity.com/paper/2604.08633/) (2026) ## Built on by (most cited first, 2 of 2) - [What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects](https://mlsanity.com/paper/2610.05533/) (2026) - [When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark](https://mlsanity.com/paper/2610.16067/) (2026) ## Abstract (arXiv) Tool-using agents are entering settings where a wrong action carries real cost, and the benchmarks certifying them grade what each simulated tool call reports having done, assuming the tool did what its interface advertises. The audit taxonomies we survey publish no category for that assumption, and a defect beneath a score is present on every rerun. We treat a tool's advertised surfaces as an executable contract, check the implementation against it, and trace each score's provenance through the task files and evaluator code to the verdicts that derive from state a defective tool should have written. Across 34 audited mutating tools in four benchmarks we confirm seven tool defects and one evaluator property at pinned commits. On injected defects the checker raised no false positive in 25 flags, flagged 2 of 5 negative controls, and missed most: in 29 of 33 scored misses a clause covered the defect but no probe revealed it. The checker's own static half, run alone, flags 14 of 17 confirmed sites, so on these findings the dynamic half confirms and traces rather than discovers. Twelve further AgentDojo tools, with six held-out tools and the seven audited first, complete its 25-tool mutating surface, on which at least 5 tools diverge from their advertised surface as our contracts read it, a rate for AgentDojo alone. No gold trajectory reaches either tau2-bench defect; on 1,120 paths built to isolate the telecom defect, a number fixed by construction, the evaluator rewards a refuel of a suspended line and fails the repaired tool. The clearest case is a clinical benchmark whose tool tells the agent each write executed under a documented no-write design its interface does not disclose; its grader takes that message as evidence, so its action success rate records whether a request carried the expected payload, not whether any record changed.