# DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents

arXiv: https://arxiv.org/abs/2601.20975 · HTML: https://mlsanity.com/paper/2601.20975/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 33 of its 39 references that are in the map.
- First posted on arXiv 28 Jan 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 33)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [Long-form factuality in large language models](https://mlsanity.com/paper/2403.18802/) (2024)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [CRAG -- Comprehensive RAG Benchmark](https://mlsanity.com/paper/2406.04744/) (2024)
- [LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores](https://mlsanity.com/paper/2311.09766/) (2023)
- [FELM: Benchmarking Factuality Evaluation of Large Language Models](https://mlsanity.com/paper/2310.00741/) (2023)
- [SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge](https://mlsanity.com/paper/2509.07968/) (2025)
- [Search Arena: Analyzing Search-Augmented LLMs](https://mlsanity.com/paper/2506.05334/) (2025)
- [WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries](https://mlsanity.com/paper/2407.17468/) (2024)
- [Agentic Information Retrieval](https://mlsanity.com/paper/2410.09713/) (2024)
- [MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents](https://mlsanity.com/paper/2508.11133/) (2025)
- [The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality](https://mlsanity.com/paper/2512.10791/) (2025)
- [LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation](https://mlsanity.com/paper/2309.12455/) (2023)
- [FactAlign: Long-form Factuality Alignment of Large Language Models](https://mlsanity.com/paper/2410.01691/) (2024)

## Built on by (most cited first, 30 of 50)

- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [Kimi K3: Open Frontier Intelligence](https://mlsanity.com/paper/2607.24653/) (2026)
- [MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verification](https://mlsanity.com/paper/2603.15726/) (2026)
- [Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2604.11753/) (2026)
- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [AI co-mathematician: Accelerating mathematicians with agentic AI](https://mlsanity.com/paper/2605.06651/) (2026)
- [Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design](https://mlsanity.com/paper/2603.28376/) (2026)
- [JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution](https://mlsanity.com/paper/2608.25593/) (2026)
- [DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity](https://mlsanity.com/paper/2602.11685/) (2026)
- [RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards](https://mlsanity.com/paper/2605.10899/) (2026)
- [Self-Compacting Language Model Agents](https://mlsanity.com/paper/2606.23525/) (2026)
- [ACM: Agentic Context Management for Long Horizon Tasks](https://mlsanity.com/paper/2607.23809/) (2026)
- [Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?](https://mlsanity.com/paper/2604.10547/) (2026)
- [When Is Enough Not Enough? Illusory Completion in Search Agents](https://mlsanity.com/paper/2602.07549/) (2026)
- [Interactive Benchmarks](https://mlsanity.com/paper/2603.04737/) (2026)
- [Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents](https://mlsanity.com/paper/2603.18516/) (2026)
- [DR-Venus: Towards Frontier Edge-Scale Deep Research Agents with Only 10K Open Data](https://mlsanity.com/paper/2604.19859/) (2026)
- [LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?](https://mlsanity.com/paper/2605.28721/) (2026)
- [AREX: Towards a Recursively Self-Improving Agent for Deep Research](https://mlsanity.com/paper/2607.21461/) (2026)
- [LoHoSearch: Benchmarking Long-Horizon Search Agents Beyond the Human Difficulty Ceiling](https://mlsanity.com/paper/2606.12837/) (2026)
- [EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledge](https://mlsanity.com/paper/2606.13120/) (2026)
- [Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems](https://mlsanity.com/paper/2605.04018/) (2026)
- [Deep Reasoning in General Purpose Agents via Structured Meta-Cognition](https://mlsanity.com/paper/2605.11388/) (2026)
- [Dr-CiK: A Benchmark testing Deep Research for Context-Aided Forecasting](https://mlsanity.com/paper/2605.27904/) (2026)
- [AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities](https://mlsanity.com/paper/2607.13705/) (2026)
- [Hunt Globally: Wide Search AI Agents for Drug Asset Scouting in Investing, Business Development, and Competitive Intelligence](https://mlsanity.com/paper/2602.15019/) (2026)
- [VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild](https://mlsanity.com/paper/2605.27882/) (2026)
- [FineVerify: Scaling Test-Time Compute with Fine-Grained Self-Verification for Agentic Search](https://mlsanity.com/paper/2606.00660/) (2026)
- [Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL](https://mlsanity.com/paper/2607.16204/) (2026)
- [A2E : An End-to-End Agent Auditing Engine](https://mlsanity.com/paper/2608.07346/) (2026)

## Abstract (arXiv)

We introduce DeepSearchQA, a 900-prompt benchmark for evaluating agents on difficult multi-step information-seeking tasks across 17 different fields. Unlike traditional benchmarks that target single answer retrieval or broad-spectrum factuality, DeepSearchQA features a dataset of challenging, handcrafted tasks designed to evaluate an agent's ability to execute complex search plans to generate exhaustive answer lists. This shift in design explicitly tests three critical, yet under-evaluated capabilities: 1) systematic collation of fragmented information from disparate sources, 2) de-duplication and entity resolution to ensure precision, and 3) the ability to reason about stopping criteria within an open-ended search space. Each task is structured as a causal chain, where discovering information for one step is dependent on the successful completion of the previous one, stressing long-horizon planning and context retention. All tasks are grounded in the open web with objectively verifiable answer sets. Our comprehensive evaluation of state-of-the-art agent architectures reveals significant performance limitations: even the most advanced models struggle to balance high recall with precision. We observe distinct failure modes ranging from premature stopping (under-retrieval) to hedging behaviors, where agents cast an overly wide net of low-confidence answers to artificially boost recall. These findings highlight critical headroom in current agent designs and position DeepSearchQA as an essential diagnostic tool for driving future research toward more robust, deep-research capabilities.