mlsanityData to 7 Oct 2026

When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation

arXiv 2608.14940 ↗First posted 14 Aug 2026cs.AI cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

Agent evaluations commonly score the state observed when a run stops and count the run as one trial. Interpreting that score as a final result from a separate trial requires outcome finality and cross-unit separation. Outcome finality requires that later events cannot change the claimed result, while cross-unit separation requires that earlier runs cannot change the relevant conditions of later ones. The endpoint establishes neither condition by itself, and the two can hold independently. Waiting for a delayed outcome may settle the label even though its state remains available to another run. Isolation may prevent carryover even though the scored outcome remains unresolved. We develop a completion argument that identifies the evidence needed for each decision. A final success or failure label is justified only when every relevant effect is resolved or bounded tightly enough to fix the outcome. Any remaining uncertainty must be reported. First, in a controlled replay with fixed agent actions, we find that endpoint and terminal labels differ for every nonzero-delay operation and that a delayed write changes the next run's score under shared state but has no such effect after namespacing or verified reset. Second, in a review of ten public protocols, we find that reset or deliberate retention is documented explicitly more often than unfinished operations or evidence for separate scoring. Finally, we propose an open-effects record for operations and resources that may remain relevant after the endpoint, their status, and their possible effects on the scored outcome or another run.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 14 Aug 2026.

Builds on

28 of 41 references.

References · 28

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  5. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  6. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  7. 2019Measurement and Fairnesscited by 299
  8. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  9. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  10. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  11. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  12. 2017Time Limits in Reinforcement Learningcited by 72
16 more
  1. 2026Cordon: Semantic Transactions for Tool-Using LLM Agentscited by 19
  2. 2025Scaling Agents for Computer Usecited by 14
  3. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJackcited by 13
  4. 2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 10
  5. 2026AgentCanary: A Security Evaluation Framework for Autonomous AI Agents in Real Executable Environmentscited by 10
  6. 2025Audit Cards: Contextualizing AI Evaluationscited by 7
  7. 2026Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agentscited by 7
  8. 2026What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schemacited by 6
  9. 2026Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AIcited by 6
  10. 2024Declare and Justify: Explicit assumptions in AI evaluations are necessary for effective regulationcited by 4
  11. 2026Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluationcited by 4
  12. 2026Training Agents to Self-Report Misbehaviorcited by 3
  13. 2026Interactive Evaluation Requires a Design Sciencecited by 3
  14. 2026AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarioscited by 3
  15. 2026How Benchmarks Mis-Score Computer-Use Agentscited by 3
  16. 2026Engagement Process: Rethinking the Temporal Interface of Action and Observationcited by 2

Built on by

Not cited yet.