Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Search
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
Retrieval systems are trained and evaluated on a static idea of usefulness: hand a document and a question to a reader model, see whether the answer improves, and score the document accordingly. The idea holds up when a document is read on its own. It breaks when a language model works as a search agent, issuing several queries and reasoning across turns, because a document can matter for what it lets the agent do next rather than for what it says about the current question. We measure that gap rather than argue it. Using a ReAct style agent over HotpotQA, we replay 1000 development questions and, for every document the agent read, delete it and re-run the rest of the trajectory from that point. Comparing the original run against its counterfactual gives a Counterfactual Trajectory Utility (CTU) score from three deltas: final answer quality, next query retrieval quality, and turn count. Crossing CTU against Static RAG Utility (SRU) over 23,322 document observations, the two are close to statistically independent (Spearman rho = -0.026). Roughly a third of the documents the agent reads are causally load bearing while looking useless to a static reader; we call these bridge documents. The pattern survives when the reader based axis is swapped for a BM25 and cross encoder proxy, giving a bridge cell of 27.2% on an evenly spread axis. A second experiment pins down the mechanism. Using the Observable Entity Relevance (OER) measure from prior work, entities that discriminate relevant from non-relevant candidates appear in the agent's next query 4.02 times more often than entities found only in non-relevant documents (6.1% vs 1.5%, n = 227,139). A bridge document earns its keep by handing the agent a discriminative entity that redirects the search. Static relevance and causal usefulness are different quantities in agentic retrieval, and optimizing the first does not deliver the second.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 8 of its 15 references that are in the map. First posted on arXiv 16 Jul 2026.
Builds on
8 of 15 references.
References · 8
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
- 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
- 2019Passage Re-ranking with BERTcited by 629
- 2026Entity Labels Are Not Entity Signals: A Framework for Observable Relevance in Document Re-Rankingcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections
- 2026Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG
- 2026Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads
- 2025RAVine: Reality-Aligned Evaluation for Agentic Search
- 2026GRASP: GRanularity-Aware Search Policy for Agentic RAG
- 2025Am I on the Right Track? What Can Predicted Query Performance Tell Us about the Search Behaviour of Agentic RAG
- 2026Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems
- 2026Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?