# SmartSearch: Process Reward-Guided Query Refinement for Search Agents

arXiv: https://arxiv.org/abs/2601.04888 · HTML: https://mlsanity.com/paper/2601.04888/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 43 of its 64 references that are in the map.
- First posted on arXiv 8 Jan 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 43)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [ZeroSearch: Incentivize the Search Capability of LLMs without Searching](https://mlsanity.com/paper/2505.04588/) (2025)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research](https://mlsanity.com/paper/2405.13576/) (2024)
- [Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL](https://mlsanity.com/paper/2508.07976/) (2025)
- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2305.06984/) (2023)
- [Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning](https://mlsanity.com/paper/2505.14069/) (2025)
- [OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning](https://mlsanity.com/paper/2502.11271/) (2025)

## Built on by (most cited first, 14 of 14)

- [OVD: On-policy Verbal Distillation](https://mlsanity.com/paper/2601.21968/) (2026)
- [Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration](https://mlsanity.com/paper/2602.03647/) (2026)
- [IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning](https://mlsanity.com/paper/2604.15148/) (2026)
- [When Is Enough Not Enough? Illusory Completion in Search Agents](https://mlsanity.com/paper/2602.07549/) (2026)
- [Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis](https://mlsanity.com/paper/2604.24198/) (2026)
- [ProRAG: Process-Supervised Reinforcement Learning for Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.21912/) (2026)
- [Effective Reinforcement Learning for Agentic Search by Recycling Zero-Variance Queries During Training](https://mlsanity.com/paper/2606.10709/) (2026)
- [RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents](https://mlsanity.com/paper/2605.26352/) (2026)
- [Beyond Parallel Sampling: Diverse Query Initialization for Agentic Search](https://mlsanity.com/paper/2606.17209/) (2026)
- [STEC: Evidence Compression for Deep Search in Open-domain Multi-Hop QA](https://mlsanity.com/paper/2607.10795/) (2026)
- [HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning](https://mlsanity.com/paper/2608.01597/) (2026)
- [IGSD: Environment-Verified Hindsight Self-Distillation for Search Agents](https://mlsanity.com/paper/2609.32694/) (2026)
- [SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents](https://mlsanity.com/paper/2610.13157/) (2026)
- [WeFair: Fidelity-Aware Alignment from Human Feedback for Personalized Query Rewriting in Short-Video Search](https://mlsanity.com/paper/2610.16927/) (2026)

## Abstract (arXiv)

Large language model (LLM)-based search agents have proven promising for addressing knowledge-intensive problems by incorporating information retrieval capabilities. Existing works largely focus on optimizing the reasoning paradigms of search agents, yet the quality of intermediate search queries during reasoning remains overlooked. As a result, the generated queries often remain inaccurate, leading to unexpected retrieval results and ultimately limiting search agents' overall effectiveness. To mitigate this issue, we introduce SmartSearch, a framework built upon two key mechanisms: (1) Process rewards, which provide fine-grained supervision for the quality of each intermediate search query through Dual-Level Credit Assessment. (2) Query refinement, which promotes the optimization of query generation by selectively refining low-quality search queries and regenerating subsequent search rounds based on these refinements. To enable the search agent to progressively internalize the ability to improve query quality under the guidance of process rewards, we design a three-stage curriculum learning framework. This framework guides the agent through a progression from imitation, to alignment, and ultimately to generalization. Experimental results show that SmartSearch consistently surpasses existing baselines, and additional quantitative analyses further confirm its significant gains in both search efficiency and query quality. The code is available at https://github.com/MYVAE/SmartSearch.