mlsanityData to 7 Oct 2026

How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks

arXiv 2607.12338 ↗First posted 14 Jul 2026cs.AIcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Agent benchmarks often compare two agents after all tasks have run, but costly evaluations make partial runs tempting. A task fraction alone does not show whether a partial run supports the same pairwise conclusion as the completed benchmark. We study this question by replaying completed public task-level records from SWE-bench, AppWorld, and tau-bench. A partial budget counts as enough only when it supports the completed benchmark's decision, covers required task groups, and leaves no more than a target fraction of comparisons unresolved. The required task fraction varies sharply. At the strict 0 percentage point threshold on a 5 percentage point budget grid, AppWorld first meets all targets at 15 percent, tau-bench at 25 percent, and SWE-bench Verified at 90 percent; SWE-bench Lite does not meet all targets by 95 percent under the primary coverage rule. Partial-evaluation reports should state how much one agent must outperform another, how tasks are selected, what coverage rule is required, what decision rule is used, and how many comparisons may remain unresolved.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 10 of its 18 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

10 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun

Other references · 9

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  3. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  4. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  5. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  6. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 220
  7. 2024Examining the robustness of LLM evaluation to the distributional assumptions of benchmarkscited by 31
  8. 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 16
  9. 2025Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluationcited by 0
  2. 2026Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision
  2. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarks
  3. 2026The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean
  4. 2026The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?
  5. 2026What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
  6. 2026Grounded Checklist Partial Credit for Agent Skill Trajectories
  7. 2026Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
  8. 2026Benchmark Everything Everywhere All at Once