mlsanityData to 7 Oct 2026

DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning

arXiv 2510.09255 ↗First posted 10 Oct 2025cs.CLcited by 0

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 1 more
Abstract (arXiv)

Enhancing LLMs with the ability to actively search external knowledge is crucial for complex and real-world tasks. Current approaches either rely on prompting to elicit the model's innate agent capabilities, or suffer from performance ceilings and collapse when applying RL to complex interactive tasks, leaving their true agentic potential untapped. To address this, we introduce Dynamic-filter Sequence-level Policy Optimization (DSPO), an improved RL algorithm designed for robust agent training through sequence-level optimization and dynamic sample filtering. We train our model purely through RL to interleave multi-turn search and reasoning, obviating the need for supervised demonstration data. Across multiple QA benchmarks, our 7B model improves over a comparable previous work by 34.1%, and even outperforms the 14B model from previous work in complex multihop QA such as HotpotQA by nearly 9% relative, maintaining exceptional training stability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 39 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

23 of 39 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents21 May
  2. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning25 Mar
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning12 Mar
  4. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning7 Mar
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  10. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
6 more
  1. 2025Group Sequence Policy Optimizationcited by 788
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  4. 2025Geometric-Mean Policy Optimizationcited by 107
  5. 2025Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Toolscited by 106
  6. 2024Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?cited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search
  2. 2025Agentic Reinforced Policy Optimization
  3. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL
  4. 2025Demystifying Reinforcement Learning in Agentic Reasoning
  5. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
  6. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
  7. 2026PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent
  8. 2026KARL: Knowledge Agents via Reinforcement Learning