mlsanityData to 7 Oct 2026

DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL

arXiv 2509.10446 ↗First posted 12 Sep 2025cs.CLcited by 53

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Augmenting large language models (LLMs) with browsing tools substantially improves their potential as deep search agents to solve complex, real-world tasks. Yet, open LLMs still perform poorly in such settings due to limited long-horizon reasoning capacity with browsing tools and the lack of sufficiently difficult supervised data. To address these challenges, we present DeepDive to advance deep search agents. First, we propose a strategy to automatically synthesize complex, difficult, and hard-to-find questions from open knowledge graphs. Second, we apply end-to-end multi-turn reinforcement learning (RL) to enhance LLMs' long-horizon reasoning with deep search. To encourage diversity and reduce redundancy, we design a redundancy penalty that discourages repeated similar queries. Experiments show that DeepDive-32B achieves a new open-source competitive result on BrowseComp, outperforming WebSailor, DeepSeek-R1-Browse, and Search-o1. We demonstrate that multi-turn RL training improves deep search ability and significantly contributes to the performance improvements across multiple benchmarks. We observe that DeepDive enables test-time scaling of tool calls and parallel sampling. All datasets, models, and code are publicly available at https://github.com/THUDM/DeepDive.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 39 of its 54 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

39 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents8 Sep
  2. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL11 Aug
  3. 2025WebDancer: Towards Autonomous Information Seeking Agency28 May
  4. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Models9 Jan

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2024Qwen2.5 Technical Reportcited by 4,713
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
23 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  4. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  6. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  7. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  8. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  9. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  10. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  11. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  12. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  13. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  14. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  15. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  16. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  17. 2025BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinesecited by 139
  18. 2025WebShaper: Agentically Data Synthesizing via Information-Seeking Formalizationcited by 118
  19. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  20. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 65
  21. 2025SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesiscited by 55
  22. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 33
  23. 2025Agent models: Internalizing Chain-of-Action Generation into Reasoning modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 40.

53 papers.

Sort:

By area:

  1. 2025AgentFold: Long-Horizon Web Agents with Proactive Context Managementcited by 95
  2. 2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 59
  3. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
  4. 2025Scaling Agents via Continual Pre-trainingcited by 39
  5. 2026OpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Datacited by 32
  6. 2026MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agentscited by 21
  7. 2025Beyond Turn Limits: Training Deep Search Agents with Dynamic Context Windowcited by 19
  8. 2026Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Taskscited by 18
  9. 2025WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seekingcited by 17
  10. 2025ParallelMuse: Agentic Parallel Thinking for Deep Information Seekingcited by 14
  11. 2026Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewardscited by 14
  12. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
38 more
  1. 2026OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agentscited by 10
  2. 2026RE-TRAC: REcursive TRAjectory Compression for Deep Search Agentscited by 9
  3. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  4. 2026HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agentscited by 8
  5. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  6. 2025InfoAgent: Advancing Autonomous Information-Seeking Agentscited by 7
  7. 2025A2FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoningcited by 6
  8. 2026DEEPMED: Building a Medical DeepResearch Agent via Multi-hop Med-Search Data and Turn-Controlled Agentic Training & Inferencecited by 6
  9. 2026GISA: A Benchmark for General Information-Seeking Assistantcited by 6
  10. 2026MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMscited by 6
  11. 2025Nested Browser-Use Learning for Agentic Information Seekingcited by 5
  12. 2026SRR-Judge: Step-Level Rating and Refinement for Enhancing Search-Integrated Reasoning in Search Agentscited by 5
  13. 2026DR-Venus: Towards Frontier Edge-Scale Deep Research Agents with Only 10K Open Datacited by 5
  14. 2026Recursive Agent Optimizationcited by 4
  15. 2026Rethinking Continual Experience Internalization for Self-Evolving LLM Agentscited by 4
  16. 2026FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agentscited by 4
  17. 2025Synthesizing Agentic Data for Web Agents with Progressive Difficulty Enhancement Mechanismscited by 3
  18. 2026ICA: Information-Aware Credit Assignment for Visually Grounded Long-Horizon Information-Seeking Agentscited by 3
  19. 2026STAMP: Provenance-Guided Credit Assignment for Deep Search Agentscited by 3
  20. 2026CoEvoKG: Co-Evolving Knowledge Graphs with Self-Evolving Search Agentscited by 3
  21. 2026ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budgetcited by 2
  22. 2026EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledgecited by 2
  23. 2026SearchMaster: Grounded and Regulated Self-Play for Search Agentscited by 2
  24. 2026ScholarGym: Benchmarking Large Language Model Capabilities in the Information-Gathering Stage of Deep Researchcited by 1
  25. 2026GraphScout: Empowering Large Language Models with Intrinsic Exploration Ability for Agentic Graph Reasoningcited by 1
  26. 2026LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewardscited by 1
  27. 2026SAGE: Steerable Agentic Data Generation for Deep Search with Execution Feedbackcited by 0
  28. 2026Mind DeepResearch Technical Reportcited by 0
  29. 2026TreeSeeker: Tree-Structured Trial, Error, and Return in Deep Searchcited by 0
  30. 2026Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learningcited by 0
  31. 2026DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environmentcited by 0
  32. 2026AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphscited by 0
  33. 2026Distilling Temporal Search and Reasoning: Evolving LLMs for Future Prediction via Harness-Assisted Efficient Data Synthesiscited by 0
  34. 2026G-ReAct: Graph-Guided Deep Search via Structure-State Co-Evolutioncited by 0
  35. 2026Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agentscited by 0
  36. 2026Boosting Deepresearch and LongContext Ability with Self-Generated Deepresearch Rollouts Tracescited by 0
  37. 2026IterSynth: Rethinking Deep Search Agents via Role-Decoupled Iterative Synthesiscited by 0
  38. 2026Dr. Free: You Don't Need Difficulty Rewards for Self-Evolving Search Agentscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learning
  2. 2026Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
  3. 2026OpenSeeker-v2: Pushing the Limits of Search Agents with Informative and High-Difficulty Trajectories
  4. 2025ManuSearch: Democratizing Deep Search in Large Language Models with a Transparent and Open Multi-Agent Framework
  5. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search
  6. 2025Reinforcement Learning for Long-Horizon Multi-Turn Search Agents
  7. 2026LexiHorizon: Stabilizing Reinforcement Learning for Long-Horizon Deep Search
  8. 2025R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning