mlsanityData to 14 Oct 2026

StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization

arXiv 2505.15107 ↗First posted 21 May 2025cs.CL cs.AI cs.IRcited by 88

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Efficient multi-hop reasoning requires Large Language Models (LLMs) based agents to acquire high-value external knowledge iteratively. Previous work has explored reinforcement learning (RL) to train LLMs to perform search-based document retrieval, achieving notable improvements in QA performance, but underperform on complex, multi-hop QA resulting from the sparse rewards from global signal only. To address this gap in existing research, we introduce StepSearch, a framework for search LLMs that trained with step-wise proximal policy optimization method. It consists of richer and more detailed intermediate search rewards and token-level process supervision based on information gain and redundancy penalties to better guide each search step. We constructed a fine-grained question-answering dataset containing sub-question-level search trajectories based on open source datasets through a set of data pipeline method. On standard multi-hop QA benchmarks, it significantly outperforms global-reward baselines, achieving 11.2% and 4.2% absolute improvements for 3B and 7B models over various search with RL baselines using only 19k training data, demonstrating the effectiveness of fine-grained, stepwise supervision in optimizing deep search LLMs. Our code will be released on https://github.com/Zillwang/StepSearch.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 59 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

43 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning12 Mar
  2. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning7 Mar
  3. 2025Chain-of-Retrieval Augmented Generation24 Jan
  4. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Models9 Jan
  5. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions20 Dec

Other references · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  12. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
26 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  3. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  4. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  5. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  6. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  7. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  8. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  9. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  10. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 249
  11. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  12. 2024Tool Learning with Large Language Models: A Surveycited by 223
  13. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  14. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  15. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  16. 2024Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrieverscited by 84
  17. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directionscited by 70
  18. 2025DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learningcited by 66
  19. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agentscited by 56
  20. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approachescited by 43
  21. 2025RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learningcited by 30
  22. 2025Process-Supervised Reinforcement Learning for Code Generationcited by 22
  23. 2025Uncertainty-Aware Step-wise Verification with Generative Reward Modelscited by 22
  24. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 21
  25. 2024Multi-Reranker: Maximizing performance of retrieval-augmented generation in the FinanceRAG challengecited by 10
  26. 2024PRACT: Optimizing Principled Reasoning and Acting of LLM Agentcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 61.

88 papers.

Sort:

By area:

  1. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 278
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2026SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalizationcited by 94
  5. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 80
  6. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  7. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 72
  8. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
  9. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 34
  10. 2025ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learningcited by 26
  11. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  12. 2025Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learningcited by 17
38 more
  1. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 16
  2. 2025On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacementcited by 15
  3. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  4. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learningcited by 13
  5. 2026RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewardscited by 12
  6. 2025GAP: Graph-Based Agent Planning with Parallel Tool Use and Reinforcement Learningcited by 11
  7. 2025InfoFlow: Reinforcing Search Agent Via Reward Density Optimizationcited by 11
  8. 2026OVD: On-policy Verbal Distillationcited by 11
  9. 2026Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnessescited by 10
  10. 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimizationcited by 9
  11. 2026IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoningcited by 9
  12. 2026Harnessing LLM Agents with Skill Programscited by 9
  13. 2026Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoningcited by 9
  14. 2025QAgent: A modular Search Agent with Interactive Query Understandingcited by 8
  15. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  16. 2026SE-Search: Self-Evolving Search Agent via Memory and Dense Rewardcited by 8
  17. 2026Meta-Reinforcement Learning with Self-Reflection for Agentic Searchcited by 8
  18. 2025TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Frameworkcited by 7
  19. 2026OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Searchcited by 7
  20. 2026π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Datacited by 7
  21. 2026Co-Evolving Skill Generation and Policy Optimizationcited by 7
  22. 2026TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agentscited by 7
  23. 2025Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMscited by 6
  24. 2025Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generationcited by 6
  25. 2026E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoningcited by 6
  26. 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignmentcited by 6
  27. 2025From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMscited by 5
  28. 2025RE-Searcher: Robust Agentic Search with Goal-oriented Planning and Self-reflectioncited by 5
  29. 2025Beyond the limitation of a single query: Train your LLM for query expansion with Reinforcement Learningcited by 5
  30. 2025Thinker: Training LLMs in Hierarchical Thinking for Deep Search via Multi-Turn Interactioncited by 5
  31. 2026Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agentscited by 5
  32. 2026SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoningcited by 5
  33. 2025PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoningcited by 4
  34. 2025ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answeringcited by 4
  35. 2025Rethinking the Design of Reinforcement Learning-Based Deep Research Agentscited by 4
  36. 2026ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agentscited by 4
  37. 2026BranPO: Scalable Contrastive Branch Sampling for Long-Horizon Agentic Reinforcement Learningcited by 3
  38. 2026RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Modelscited by 3

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
  2. 2025FrugalRAG: Less is More in RL Finetuning for Multi-Hop Question Answering
  3. 2026Search-GRT: Guided Retrieval Training of Search Agents to Optimize for Complex Question Answering
  4. 2025AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning
  5. 2026PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation
  6. 2025R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
  7. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents
  8. 2025GlobalRAG: Enhancing Global Reasoning in Multi-hop Question Answering via Reinforcement Learning