# Benchmarking Deep Search over Heterogeneous Enterprise Data

arXiv: https://arxiv.org/abs/2506.23139 · HTML: https://mlsanity.com/paper/2506.23139/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 29 Jun 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)
- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://mlsanity.com/paper/2401.15391/) (2024)
- [From RAG to Memory: Non-Parametric Continual Learning for Large Language Models](https://mlsanity.com/paper/2502.14802/) (2025)
- [WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks](https://mlsanity.com/paper/2407.05291/) (2024)
- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments](https://mlsanity.com/paper/2411.02305/) (2024)
- [WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting](https://mlsanity.com/paper/2405.00823/) (2024)
- [CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions](https://mlsanity.com/paper/2505.18878/) (2025)
- [Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective](https://mlsanity.com/paper/2410.12812/) (2024)
- [MHTS: Multi-Hop Tree Structure Framework for Generating Difficulty-Controllable QA Datasets for RAG Evaluation](https://mlsanity.com/paper/2504.08756/) (2025)

## Built on by (most cited first, 23 of 23)

- [A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges](https://mlsanity.com/paper/2508.05668/) (2025)
- [SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents](https://mlsanity.com/paper/2509.06283/) (2025)
- [A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications](https://mlsanity.com/paper/2510.16724/) (2025)
- [Enterprise Deep Research: Steerable Multi-Agent Deep Research for Enterprise Analytics](https://mlsanity.com/paper/2510.17797/) (2025)
- [CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agents](https://mlsanity.com/paper/2604.21308/) (2026)
- [Don't Stop Early: Scalable Enterprise Deep Research with Controlled Information Flow and Evidence-Aware Termination](https://mlsanity.com/paper/2604.24978/) (2026)
- [Topo-RAG: Topology-aware retrieval for hybrid text-table documents](https://mlsanity.com/paper/2601.10215/) (2026)
- [ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering](https://mlsanity.com/paper/2608.10679/) (2026)
- [CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases](https://mlsanity.com/paper/2608.27391/) (2026)
- [Policy Compliance of User Requests in Natural Language for AI Systems](https://mlsanity.com/paper/2603.00369/) (2026)
- [Hypergraph Enterprise Agentic Reasoner over Heterogeneous Business Systems](https://mlsanity.com/paper/2605.14259/) (2026)
- [X-SYNTH: Beyond Retrieval -- Enterprise Context Synthesis from Observed Digital Human Attention](https://mlsanity.com/paper/2605.15505/) (2026)
- [MosaicLeaks:Privacy Risks in Querying-in-the-Open for Deep Research Agents](https://mlsanity.com/paper/2605.30727/) (2026)
- [Dual-Confidence Contrastive Decoding for Retrieval-Augmented Generation](https://mlsanity.com/paper/2607.00570/) (2026)
- [WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing](https://mlsanity.com/paper/2607.25765/) (2026)
- [PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming](https://mlsanity.com/paper/2608.15931/) (2026)
- [MegaMem: A Retrieval Solution for Ultra-Large Context Windows](https://mlsanity.com/paper/2608.22137/) (2026)
- [Benchmarking Hybrid Deep Research Across Database Querying and Web Search](https://mlsanity.com/paper/2609.09410/) (2026)
- [WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation](https://mlsanity.com/paper/2609.12171/) (2026)
- [One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extraction](https://mlsanity.com/paper/2609.21626/) (2026)
- [EvidenT: Building Trustworthy Enterprise Assistants through Evidence Groundedness and Traceability](https://mlsanity.com/paper/2609.22537/) (2026)
- [Evaluating the accuracy of KV cache reuse techniques](https://mlsanity.com/paper/2609.31415/) (2026)
- [Follow the Entities: A Corpus Map for Agentic Search](https://mlsanity.com/paper/2609.37226/) (2026)

## Abstract (arXiv)

We present a new benchmark for evaluating Deep Search--a realistic and complex form of retrieval-augmented generation (RAG) that requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources. These include documents, meeting transcripts, Slack messages, GitHub, and URLs, which vary in structure and often contain human-to-human interactions. We build it using a synthetic data pipeline that simulates business workflows across product planning, development, and support stages, generating interconnected content with realistic noise and multi-hop questions with guaranteed ground-truth answers. We release our benchmark with both answerable and unanswerable queries, and retrieval pool of 39,190 enterprise artifacts, enabling fine-grained evaluation of long-context LLM and RAG systems. Our experiments reveal that even the best-performing agentic RAG methods achieve an average performance score of 32.96 on our benchmark. With further analysis, we highlight retrieval as the main bottleneck: existing methods struggle to conduct deep searches and retrieve all necessary evidence. Consequently, they often reason over partial context, leading to significant performance degradation.