Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation
Not on the map. Why?
Abstract (arXiv)
A correct answer can conceal why an agent succeeded. Once agents change their information state during evaluation, correctness no longer distinguishes intended reasoning from answer acquisition. Outcome evidence and exposure detection do not establish whether success depended on an acquired target; we call this missing evaluation object success provenance. AcquaBench audits it through matched CLEAN, GOLD, and SHAM value substitution on four standardized surfaces with joint qid-clustered analysis. CLEAN retains benchmark-authorized information. GOLD makes the correct target available. SHAM preserves source structure and exposure opportunity but substitutes a matched incorrect value. GOLD minus CLEAN measures the total score response to correct-target availability; GOLD minus SHAM tests whether that response tracks target correctness beyond matched source exposure. In D0, GOLD exceeds SHAM by 19.1 to 25.9 percentage points, showing that success follows the correct value. In D2, GOLD still exceeds SHAM under distributed sufficiency while coloc no longer transfers as a high-score marker, with AUROC 0.376 and 0.142. Behavioral dependence can thus persist beyond this probe's intended observation unit. In model comparison, a supported 5.0-point CLEAN score gap compresses to a raw GOLD difference of -0.6 points without establishing rank inversion. Agent benchmarks should report success together with whether the evaluated information state supported it.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 27 Jul 2026.
Builds on
23 of 33 references.
References · 23
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2023GAIA: a benchmark for General AI Assistantscited by 1,128
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
- 2023Enabling Large Language Models to Generate Text with Citationscited by 595
- 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
11 more
- 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
- 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
- 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
- 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
- 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
- 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
- 2024GTA: A Benchmark for General Tool Agentscited by 77
- 2023Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noisecited by 54
- 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contaminationcited by 23
- 2026Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluationcited by 9
- 2026Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluationcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.