mlsanityData to 7 Oct 2026

Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL

arXiv 2508.07976 ↗First posted 11 Aug 2025cs.CL cs.AIcited by 144

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Recent advancements in LLM-based agents have demonstrated remarkable capabilities in handling complex, knowledge-intensive tasks by integrating external tools. Among diverse choices of tools, search tools play a pivotal role in accessing vast external knowledge. However, open-source agents still fall short of achieving expert-level Search Intelligence, the ability to resolve ambiguous queries, generate precise searches, analyze results, and conduct thorough exploration. Existing approaches fall short in scalability, efficiency, and data quality. For example, small turn limits in existing online RL methods, e.g. <=10, restrict complex strategy learning. This paper introduces ASearcher, an open-source project for large-scale RL training of search agents. Our key contributions include: (1) Scalable fully asynchronous RL training that enables long-horizon search while maintaining high training efficiency. (2) A prompt-based LLM agent that autonomously synthesizes high-quality and challenging QAs, creating a large-scale QA dataset. Through RL training, our prompt-based QwQ-32B agent achieves substantial improvements, with 78.0% and 34.3% Avg@4 gains on xBench and GAIA, respectively. Notably, our agent exhibits extreme long-horizon search, with tool calls exceeding 100 turns and output tokens exceeding 400k during training time. With a simple agent design and no external LLMs, ASearcher-Web-QwQ achieves Avg@4 scores of 51.1 on xBench and 58.7 on GAIA, surpassing existing open-source 32B agents. Finally, we also show that ASearcher-Web-QwQ could achieve performance of commercial systems using external summary tool in a zero-shot transfer manner and test-time search. We open-source our models, training data, and codes in https://github.com/inclusionAI/ASearcher.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 35 of its 45 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

35 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025WebSailor: Navigating Super-human Reasoning for Web Agent3 Jul
  2. 2025WebDancer: Towards Autonomous Information Seeking Agency28 May
  3. 2025Open Deep Search: Democratizing Search with Open-source Reasoning Agents26 Mar
  4. 2025ReAgent: Reversible Multi-Agent Reasoning for Knowledge-Enhanced Multi-Hop QA10 Mar
  5. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Models9 Jan

Other references · 30

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  11. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  12. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
18 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  3. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  4. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  5. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  6. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  7. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  8. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  9. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  10. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  11. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  12. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  13. 2025SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesiscited by 55
  14. 2023Calibrating LLM-Based Evaluatorcited by 53
  15. 2024Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Modelscited by 41
  16. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 39
  17. 2025INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learningcited by 24
  18. 2025RAG-R1: Incentivizing the Search and Reasoning Capabilities of LLMs through Multi-query Parallelismcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 52; 2026: 92.

144 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 132
  4. 2025AgentFold: Long-Horizon Web Agents with Proactive Context Managementcited by 95
  5. 2025WebExplorer: Explore and Evolve for Training Long-Horizon Web Agentscited by 86
  6. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  7. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  8. 2025Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Taskscited by 62
  9. 2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 59
  10. 2025DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RLcited by 53
  11. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agentcited by 44
  12. 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 42
38 more
  1. 2025Scaling Agents via Continual Pre-trainingcited by 39
  2. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  3. 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agentscited by 35
  4. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
  5. 2026Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interactioncited by 33
  6. 2026OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesiscited by 32
  7. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  8. 2025SR-Scientist: Scientific Equation Discovery With Agentic AIcited by 30
  9. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 29
  10. 2025Single-stream Policy Optimizationcited by 27
  11. 2025Flash-Searcher: Fast and Effective Web Agents via DAG-Based Parallel Executioncited by 27
  12. 2026ExpSeek: Self-Triggered Experience Seeking for Web Agentscited by 23
  13. 2025IterResearch: Rethinking Long-Horizon Agents with Interaction Scalingcited by 22
  14. 2026Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agentscited by 22
  15. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  16. 2025Beyond Turn Limits: Training Deep Search Agents with Dynamic Context Windowcited by 19
  17. 2025WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seekingcited by 17
  18. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  19. 2026From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agentscited by 17
  20. 2026ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agentscited by 17
  21. 2025ParallelMuse: Agentic Parallel Thinking for Deep Information Seekingcited by 14
  22. 2026Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewardscited by 14
  23. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 13
  24. 2026StackPlanner: A Centralized Hierarchical Multi-Agent System with Task-Experience Memory Managementcited by 13
  25. 2026AtomMem : Learnable Dynamic Agentic Memory with Atomic Memory Operationcited by 13
  26. 2026Inference-Time Budget Control for LLM Search Agentscited by 13
  27. 2026Terminal-World: Scaling Terminal-Agent Environments via Agent Skillscited by 13
  28. 2025Pushing Test-Time Scaling Limits of Deep Search with Asymmetric Verificationcited by 12
  29. 2026SmartSearch: Process Reward-Guided Query Refinement for Search Agentscited by 12
  30. 2026AgentCPM-Explore: Realizing Long-Horizon Deep Exploration for Edge-Scale Agentscited by 12
  31. 2026Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Designcited by 12
  32. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 11
  33. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  34. 2025Repurposing Synthetic Data for Fine-grained Search Agent Supervisioncited by 10
  35. 2026OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agentscited by 10
  36. 2026Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaborationcited by 10
  37. 2026WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learningcited by 9
  38. 2026AgentSwing: Adaptive Parallel Context Management Routing for Long-Horizon Web Agentscited by 9

50 most cited of 144 All 144 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
  2. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents
  3. 2025Reinforcement Learning for Long-Horizon Multi-Turn Search Agents
  4. 2025AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning
  5. 2025SSRL: Self-Search Reinforcement Learning
  6. 2026SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task
  7. 2026KARL: Knowledge Agents via Reinforcement Learning
  8. 2026PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent