# Characterizing Deep Research: A Benchmark and Formal Definition

arXiv: https://arxiv.org/abs/2508.04183 · HTML: https://mlsanity.com/paper/2508.04183/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 27 of 27)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://mlsanity.com/paper/2504.03160/) (2025)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval](https://mlsanity.com/paper/2407.12883/) (2024)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://mlsanity.com/paper/2402.14207/) (2024)
- [OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs](https://mlsanity.com/paper/2411.14199/) (2024)
- [DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research](https://mlsanity.com/paper/2505.19253/) (2025)
- [Deep Research Bench: Evaluating AI Web Research Agents](https://mlsanity.com/paper/2506.06287/) (2025)
- [ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry](https://mlsanity.com/paper/2507.16280/) (2025)
- [Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks](https://mlsanity.com/paper/2507.23194/) (2025)
- [CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning](https://mlsanity.com/paper/2503.13517/) (2025)
- [Claim Verification in the Age of Large Language Models: A Survey](https://mlsanity.com/paper/2408.14317/) (2024)
- [DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learning](https://mlsanity.com/paper/2505.24332/) (2025)
- [Precise Information Control in Long-Form Text Generation](https://mlsanity.com/paper/2506.06589/) (2025)
- [Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents](https://mlsanity.com/paper/2507.05495/) (2025)
- [Analysis of Plan-based Retrieval for Grounded Text Generation](https://mlsanity.com/paper/2408.10490/) (2024)
- [VeriTrail: Closed-Domain Hallucination Detection with Traceability](https://mlsanity.com/paper/2505.21786/) (2025)

## Built on by (most cited first, 30 of 30)

- [ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents](https://mlsanity.com/paper/2511.07685/) (2025)
- [DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis](https://mlsanity.com/paper/2508.20033/) (2025)
- [SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents](https://mlsanity.com/paper/2509.06283/) (2025)
- [Accumulating Context Changes the Beliefs of Language Models](https://mlsanity.com/paper/2511.01805/) (2025)
- [DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation](https://mlsanity.com/paper/2512.17776/) (2025)
- [How Far Are We from Genuinely Useful Deep Research Agents?](https://mlsanity.com/paper/2512.01948/) (2025)
- [DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality](https://mlsanity.com/paper/2603.05912/) (2026)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents](https://mlsanity.com/paper/2602.21230/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Deep Research with Open-Domain Evaluation and Multi-Stage Guardrails for Safety](https://mlsanity.com/paper/2510.10994/) (2025)
- [LABELING COPILOT: A Deep Research Agent for Automated Data Curation in Computer Vision](https://mlsanity.com/paper/2509.22631/) (2025)
- [When Is Enough Not Enough? Illusory Completion in Search Agents](https://mlsanity.com/paper/2602.07549/) (2026)
- [Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks](https://mlsanity.com/paper/2603.06942/) (2026)
- [InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search](https://mlsanity.com/paper/2605.07510/) (2026)
- [Towards Efficient Agents: A Co-Design of Inference Architecture and System](https://mlsanity.com/paper/2512.18337/) (2025)
- [R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoning](https://mlsanity.com/paper/2604.03004/) (2026)
- [AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments](https://mlsanity.com/paper/2607.05174/) (2026)
- [ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents](https://mlsanity.com/paper/2512.00986/) (2025)
- [Mind2Report: Expert-Level Commercial Report Synthesis via Cognitive Deep Research Agent](https://mlsanity.com/paper/2601.04879/) (2026)
- [Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users](https://mlsanity.com/paper/2603.16120/) (2026)
- [HotelQuEST: Balancing Quality and Efficiency in Agentic Search](https://mlsanity.com/paper/2602.23949/) (2026)
- [LLM-Oriented Information Retrieval: A Denoising-First Perspective](https://mlsanity.com/paper/2605.00505/) (2026)
- [LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake](https://mlsanity.com/paper/2606.10460/) (2026)
- [From Inertia to Objectivity: Improving Deep Research Agents with Noise Isolation](https://mlsanity.com/paper/2608.23045/) (2026)
- [Dual-Confidence Contrastive Decoding for Retrieval-Augmented Generation](https://mlsanity.com/paper/2607.00570/) (2026)
- [From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution](https://mlsanity.com/paper/2608.02163/) (2026)
- [Benchmarking Hybrid Deep Research Across Database Querying and Web Search](https://mlsanity.com/paper/2609.09410/) (2026)
- [What Happens During Autonomous Deep Research After the User Steps Away?](https://mlsanity.com/paper/2609.33509/) (2026)
- [DeepRewind: Predicting and Repairing Premature Commitments in Deep Research Agents](https://mlsanity.com/paper/2609.36344/) (2026)

## Abstract (arXiv)

Information tasks such as writing surveys or analytical reports require complex search and reasoning, and have recently been grouped under the umbrella of textitdeep research -- a term also adopted by recent models targeting these capabilities. Despite growing interest, the scope of the deep research task remains underdefined and its distinction from other reasoning-intensive problems is poorly understood. In this paper, we propose a formal characterization of the deep research (DR) task and introduce a benchmark to evaluate the performance of DR systems. We argue that the core defining feature of deep research is not the production of lengthy report-style outputs, but rather the high fan-out over concepts required during the search process, i.e., broad and reasoning-intensive exploration. To enable objective evaluation, we define DR using an intermediate output representation that encodes key claims uncovered during search-separating the reasoning challenge from surface-level report generation. Based on this formulation, we propose a diverse, challenging benchmark LiveDRBench with 100 challenging tasks over scientific topics (e.g., datasets, materials discovery, prior art search) and public interest events (e.g., flight incidents, movie awards). Across state-of-the-art DR systems, F1 score ranges between 0.02 and 0.72 for any sub-category. OpenAI's model performs the best with an overall F1 score of 0.55. Analysis of reasoning traces reveals the distribution over the number of referenced sources, branching, and backtracking events executed by current DR systems, motivating future directions for improving their search mechanisms and grounding capabilities. The benchmark is available at https://github.com/microsoft/LiveDRBench.