# SE-Search: Self-Evolving Search Agent via Memory and Dense Reward

arXiv: https://arxiv.org/abs/2603.03293 · HTML: https://mlsanity.com/paper/2603.03293/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation.
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Placed in

- Retrieval-augmented generation / [Agentic RAG, agents and memory](https://mlsanity.com/rag/agentic-rag-agents-and-memory/)
- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)
- Synthetic data and self-improvement for LLMs / [Self-evolving search agents](https://mlsanity.com/synth/self-evolving-search-agents/)

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning](https://mlsanity.com/paper/2508.19828/) (2025)
- [ZeroSearch: Incentivize the Search Capability of LLMs without Searching](https://mlsanity.com/paper/2505.04588/) (2025)
- [StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization](https://mlsanity.com/paper/2505.15107/) (2025)
- [DeepAgent: A General Reasoning Agent with Scalable Toolsets](https://mlsanity.com/paper/2510.21618/) (2025)
- [MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.00332/) (2024)
- [CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic](https://mlsanity.com/paper/2511.12159/) (2025)

## Built on by (most cited first, 8 of 8)

- [Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?](https://mlsanity.com/paper/2607.17558/) (2026)
- [Improving Search Agent with One Line of Code](https://mlsanity.com/paper/2603.10069/) (2026)
- [ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget](https://mlsanity.com/paper/2604.01195/) (2026)
- [Plan Before Search: Search Agents Need Plan](https://mlsanity.com/paper/2605.28354/) (2026)
- [MEMENTO: Leveraging Web as a Learning Signal for Low-Data Domains](https://mlsanity.com/paper/2605.29795/) (2026)
- [Co-Evolving Graph and Text Memory for Training-Free Multi-Hop Question Answering](https://mlsanity.com/paper/2607.23278/) (2026)
- [Dr. Free: You Don't Need Difficulty Rewards for Self-Evolving Search Agents](https://mlsanity.com/paper/2609.33565/) (2026)
- [SEDIMA: Cross-Run Hierarchical Insight Memory for Evolutionary Search Agents](https://mlsanity.com/paper/2610.02361/) (2026)

## Abstract (arXiv)

Retrieval augmented generation (RAG) reduces hallucinations and factual errors in large language models (LLMs) by conditioning generation on retrieved external knowledge. Recent search agents further cast RAG as an autonomous, multi-turn information-seeking process. However, existing methods often accumulate irrelevant or noisy documents and rely on sparse reinforcement learning signals. We propose Self-Evolving Search, a Self-Evolving Search agent that improves online search behavior through three components, memory purification, atomic query training, and dense rewards. SE-Search follows a textitThink-Search-Memorize strategy that retains salient evidence while filtering irrelevant content. Atomic query training promotes shorter and more diverse queries, improving evidence acquisition. Dense rewards provide fine-grained feedback that speeds training. Experiments on single-hop and multi-hop question answering benchmarks show that textttSE-Search-3B outperforms strong baselines, yielding a 10.8 point absolute improvement and a 33.8% relative gain over Search-R1.footnoteWe will make the code and model weights publicly available upon acceptance.