mlsanityData to 14 Oct 2026

Where Scientific Search Agents Fail: Decision-Checkpoint Auditing of Exposure and Inspection Attempts

arXiv 2609.38670 ↗First posted 29 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Final-answer accuracy does not reveal whether a scientific-search agent failed to encounter a target paper, attempt to inspect it, or return an accepted answer after inspection. We introduce decision checkpoints that record observations and tool actions without benchmark labels during inference, then join target identities and evaluator labels to assign outcome categories from recorded events. Across five conditions on 540 answerable AutoResearchBench Deep questions in a fixed, target-enriched environment, keyword search achieves 24.6% accuracy, compared with 17.8% for raw search. The keyword condition has fewer incorrect answers with neither target exposure nor inspection, but more incorrect answers after the target is exposed and left uninspected. Compared with keyword search, read-first has 27.4% more recorded evidence-search calls. Target inspection attempts occur on 199 questions under read-first and 191 under keyword search; both conditions achieve 24.6% accuracy. The checkpoint protocol makes these question-level differences explicit, distinguishing target exposure and inspection from aggregate accuracy and total tool use.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 18 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

15 of 18 references.

References · 15

  1. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  2. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  3. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  4. 2024AI Agents That Mattercited by 219
  5. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 200
  6. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  7. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 72
  8. 2025PaSa: An LLM Agent for Comprehensive Academic Paper Searchcited by 64
  9. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 61
  10. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 50
  11. 2026Inference-Time Budget Control for LLM Search Agentscited by 13
  12. 2026Spend Less, Reason Better: Budget-Aware Value Tree Search for LLM Agentscited by 11
3 more
  1. 2026AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discoverycited by 11
  2. 2026AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Trackingcited by 9
  3. 2026SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agentscited by 2

Built on by

Not cited yet.