From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution
Not on the map. Why?
Abstract (arXiv)
Deep research benchmarks require expert-level tasks and reliable evaluation grounded in task-specific knowledge. Existing benchmarks rely heavily on expert authoring or pre-existing human-authored materials, while fully automatic construction struggles to ensure consistent and traceable verification. To address this gap, we introduce a verifiable benchmark of 500 deep research tasks spanning 31 topics and 10 major categories, with three query forms designed to probe complementary capabilities required for deep research. The benchmark is constructed automatically using an iterative Explorer-Formalizer-Challenger pipeline that progressively transforms simple questions into deep research tasks. Each task is represented as a directed acyclic graph (DAG) of atomic steps and associated checkpoints, enabling the query, DAG, and rubrics to evolve together in a controlled manner. Experiments demonstrate that the benchmark clearly discriminates among models and query types, while its fact-grounded pointwise rubrics enable fine-grained, human-aligned, and stable evaluation. Our data, implementation, and results are publicly available.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 53 references that are in the map. First posted on arXiv 3 Aug 2026.
Builds on
28 of 53 references.
References · 28
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023GAIA: a benchmark for General AI Assistantscited by 1,128
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
- 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
- 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
- 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
- 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 188
- 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
- 2025WebShaper: Agentically Data Synthesizing via Information-Seeking Formalizationcited by 121
- 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 84
16 more
- 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 76
- 2025WideSearch: Benchmarking Agentic Broad Info-Seekingcited by 69
- 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
- 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 50
- 2025ReportBench: Evaluating Deep Research Agents via Academic Survey Taskscited by 38
- 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 34
- 2025Characterizing Deep Research: A Benchmark and Formal Definitioncited by 30
- 2025DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Taskscited by 24
- 2026DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluationcited by 14
- 2025FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agentscited by 12
- 2025DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generationcited by 12
- 2026DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agentcited by 10
- 2026DR-Arena: an Automated Evaluation Framework for Deep Research Agentscited by 6
- 2026The Context Gathering Decision Process: A POMDP Framework for Agentic Searchcited by 6
- 2025ADRA-Bank: A Modular Benchmark for Academic Deep Research Agentscited by 2
- 2026AgentDisCo: Towards Disentanglement and Collaboration in Open-ended Deep Research Agentscited by 2
Built on by
Not cited yet.