Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
Deep Research agents synthesize evidence into cited reports, yet a well-cited report can still reach a misleading conclusion. Citation correctness checks whether cited sources support individual claims. It does not show whether adaptive search exposed a representative view of all documents made available for evaluation, which we call the candidate pool. Early findings redirect later queries, document choices, and stopping, so the documents an agent reads form a selective sample. Existing evaluations rarely account for this selection. We formulate the problem as adaptive evidence sampling and introduce Causal Evidence Selection Correction (CESS). CESS predicts each candidate document's evidence direction and corrects the candidate-pool average using the logged probabilities of selecting each document and reaching each search round. Shrinkage stabilizes short searches, while intervals replace point estimates when some documents cannot be sampled. We also prove that estimating the average evidence direction of a common pool differs from measuring how a change in search policy alters the evidence read. The latter requires intervention. On questions from the MS2 systematic-review benchmark, CESS reduces mean absolute error against the candidate-pool average by 9.2% and reduces the estimate's change under opposing document rankings by 39.4% relative to averaging the evidence scores of documents read. Across trajectories from a public Open Deep Research agent, the corresponding reductions reach 60.1% and 87.2%. A further 4,800 trajectories under paired interventions confirm that correcting a pool estimate and measuring a policy effect are different tasks. CESS therefore audits whether the evidence direction underlying a report reflects the documents available for evaluation, while a separate intervention analysis measures the effect of search decisions.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 25 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
17 of 25 references.
Closest ancestors · 6
Same direction, newest first
- 2026Explore Before Committing: Hypothesis-Guided Search for Deep Research Agents1 Sep
- 2026DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality6 Mar
- 2026DREAM: Deep Research Evaluation with Agentic Metrics21 Feb
- 2026Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision19 Jan
- 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge26 Jun
- 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents13 Jun
Other references · 11
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2023Enabling Large Language Models to Generate Text with Citationscited by 595
- 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
- 2019Confidence Intervals for Policy Evaluation in Adaptive Experimentscited by 95
- 2019Doubly robust off-policy evaluation with shrinkagecited by 87
- 2021Active Testing: Sample-Efficient Model Evaluationcited by 82
- 2021MS2: Multi-Document Summarization of Medical Studiescited by 80
- 2024Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversationscited by 43
- 2023Semiparametric Efficient Inference in Adaptive Experimentscited by 23
- 2026ReportLogic: Evaluating Logical Quality in Deep Research Reportscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence
- 2026DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments
- 2026When More Evidence Hurts: Publication-Bias Drift and Principled Stopping for Biomedical Causal Search
- 2026CITECHOICE: A Causal Audit of How Document Presentation Redistributes Citation Credit in Agentic Search
- 2026Benchmarking and Training Reward Models for Cross-Source Evidence Grounding in Search Agents
- 2026Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research
- 2026HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research
- 2026From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents