mlsanityData to 7 Oct 2026

Deep Research Bench: Evaluating AI Web Research Agents

arXiv 2506.06287 ↗First posted 6 May 2025cs.AIcited by 31

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Amongst the most common use cases of modern AI is LLM chat with web search enabled. However, no direct evaluations of the quality of web research agents exist that control for the continually-changing web. We introduce Deep Research Bench, consisting of 89 multi-step web research task instances of varying difficulty across 8 diverse task categories, with the answers carefully worked out by skilled humans. We provide a "RetroSearch" environment with a large frozen set of scraped web pages, and demonstrate that offline "RetroSearch" agents perform comparably to "live web" agents, enabling reliable evaluations of models over time. We provide robust agent tooling and scaffolding to benchmark major LLMs as they are released, including "thinking" models like o3 and Gemini 2.5 Pro. We include automated evaluations of the lengthy agent traces to report progress over time in hallucinations, tool use, and forgetting. Finally, we evaluate the major web research products branded as "Deep Research", "Deep Search", "Search", or "Research." Results are available on a public leaderboard at https://drb.futuresearch.ai/.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 8 of its 14 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

8 of 14 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Towards a Realistic Long-Term Benchmark for Open-Web Research Agents23 Sep
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
  3. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents4 Jul

Other references · 5

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  4. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  5. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 15.

31 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025ReportBench: Evaluating Deep Research Agents via Academic Survey Taskscited by 34
  3. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 33
  4. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 31
  5. 2025Characterizing Deep Research: A Benchmark and Formal Definitioncited by 29
  6. 2025Bench to the Future: A Pastcasting Benchmark for Forecasting Agentscited by 20
  7. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 13
  8. 2025FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agentscited by 11
  9. 2025How Far Are We from Genuinely Useful Deep Research Agents?cited by 11
  10. 2025DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidencecited by 10
  11. 2025GeoVista: Web-Augmented Agentic Visual Reasoning for Geolocalizationcited by 10
  12. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?cited by 10
19 more
  1. 2025MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipelinecited by 9
  2. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 9
  3. 2026DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytellingcited by 7
  4. 2025Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agentscited by 6
  5. 2025Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impactscited by 6
  6. 2026DR-Arena: an Automated Evaluation Framework for Deep Research Agentscited by 6
  7. 2026Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarkscited by 4
  8. 2025Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reportscited by 3
  9. 2026Automating Forecasting Question Generation and Resolution for AI Evaluationcited by 3
  10. 2026R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoningcited by 3
  11. 2025ADRA-Bank: A Modular Benchmark for Academic Deep Research Agentscited by 2
  12. 2026WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecastingcited by 2
  13. 2025Improving and Evaluating Open Deep Research Agentscited by 1
  14. 2026Evaluating Strategic Reasoning in Forecasting Agentscited by 1
  15. 2026SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoningcited by 1
  16. 2026Re2Math: Benchmarking Theorem Retrieval in Research-Level Mathematicscited by 0
  17. 2026MosaicLeaks:Privacy Risks in Querying-in-the-Open for Deep Research Agentscited by 0
  18. 2026LegalCiteTrust: Benchmarking Citation Trustworthiness in Chinese Long-Form Legal Research Reportscited by 0
  19. 2026Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responsescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
  2. 2025Deep Research: A Survey of Autonomous Research Agents
  3. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
  4. 2025WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agents
  5. 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents
  6. 2025Deep Research Agents: A Systematic Examination And Roadmap
  7. 2025DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research
  8. 2026IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents