# DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL

arXiv: https://arxiv.org/abs/2509.10446 · HTML: https://mlsanity.com/paper/2509.10446/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 39 of its 54 references that are in the map.
- First posted on arXiv 12 Sep 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 39)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://mlsanity.com/paper/2504.03160/) (2025)
- [WebDancer: Towards Autonomous Information Seeking Agency](https://mlsanity.com/paper/2505.22648/) (2025)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.19470/) (2025)

## Built on by (most cited first, 30 of 55)

- [AgentFold: Long-Horizon Web Agents with Proactive Context Management](https://mlsanity.com/paper/2510.24699/) (2025)
- [AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework](https://mlsanity.com/paper/2510.04206/) (2025)
- [WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learning](https://mlsanity.com/paper/2509.13305/) (2025)
- [Scaling Agents via Continual Pre-training](https://mlsanity.com/paper/2509.13310/) (2025)
- [OpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Data](https://mlsanity.com/paper/2603.15594/) (2026)
- [MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents](https://mlsanity.com/paper/2601.12346/) (2026)
- [Beyond Turn Limits: Training Deep Search Agents with Dynamic Context Window](https://mlsanity.com/paper/2510.08276/) (2025)
- [WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking](https://mlsanity.com/paper/2510.24697/) (2025)
- [Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2604.11753/) (2026)
- [ParallelMuse: Agentic Parallel Thinking for Deep Information Seeking](https://mlsanity.com/paper/2510.24698/) (2025)
- [Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards](https://mlsanity.com/paper/2601.06021/) (2026)
- [OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agents](https://mlsanity.com/paper/2601.18467/) (2026)
- [Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation](https://mlsanity.com/paper/2511.02303/) (2025)
- [RE-TRAC: REcursive TRAjectory Compression for Deep Search Agents](https://mlsanity.com/paper/2602.02486/) (2026)
- [HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents](https://mlsanity.com/paper/2605.07177/) (2026)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [InfoAgent: Advancing Autonomous Information-Seeking Agents](https://mlsanity.com/paper/2509.25189/) (2025)
- [FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents](https://mlsanity.com/paper/2606.12087/) (2026)
- [A2FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning](https://mlsanity.com/paper/2510.12838/) (2025)
- [Nested Browser-Use Learning for Agentic Information Seeking](https://mlsanity.com/paper/2512.23647/) (2025)
- [DEEPMED: Building a Medical DeepResearch Agent via Multi-hop Med-Search Data and Turn-Controlled Agentic Training & Inference](https://mlsanity.com/paper/2601.18496/) (2026)
- [GISA: A Benchmark for General Information-Seeking Assistant](https://mlsanity.com/paper/2602.08543/) (2026)
- [MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMs](https://mlsanity.com/paper/2602.12705/) (2026)
- [SRR-Judge: Step-Level Rating and Refinement for Enhancing Search-Integrated Reasoning in Search Agents](https://mlsanity.com/paper/2602.07773/) (2026)
- [DR-Venus: Towards Frontier Edge-Scale Deep Research Agents with Only 10K Open Data](https://mlsanity.com/paper/2604.19859/) (2026)
- [Recursive Agent Optimization](https://mlsanity.com/paper/2605.06639/) (2026)
- [STAMP: Provenance-Guided Credit Assignment for Deep Search Agents](https://mlsanity.com/paper/2607.11172/) (2026)
- [Synthesizing Agentic Data for Web Agents with Progressive Difficulty Enhancement Mechanisms](https://mlsanity.com/paper/2510.13913/) (2025)
- [ICA: Information-Aware Credit Assignment for Visually Grounded Long-Horizon Information-Seeking Agents](https://mlsanity.com/paper/2602.10863/) (2026)

## Abstract (arXiv)

Augmenting large language models (LLMs) with browsing tools substantially improves their potential as deep search agents to solve complex, real-world tasks. Yet, open LLMs still perform poorly in such settings due to limited long-horizon reasoning capacity with browsing tools and the lack of sufficiently difficult supervised data. To address these challenges, we present DeepDive to advance deep search agents. First, we propose a strategy to automatically synthesize complex, difficult, and hard-to-find questions from open knowledge graphs. Second, we apply end-to-end multi-turn reinforcement learning (RL) to enhance LLMs' long-horizon reasoning with deep search. To encourage diversity and reduce redundancy, we design a redundancy penalty that discourages repeated similar queries. Experiments show that DeepDive-32B achieves a new open-source competitive result on BrowseComp, outperforming WebSailor, DeepSeek-R1-Browse, and Search-o1. We demonstrate that multi-turn RL training improves deep search ability and significantly contributes to the performance improvements across multiple benchmarks. We observe that DeepDive enables test-time scaling of tool calls and parallel sampling. All datasets, models, and code are publicly available at https://github.com/THUDM/DeepDive.