# EvolveSearch: An Iterative Self-Evolving Search Agent

arXiv: https://arxiv.org/abs/2505.22501 · HTML: https://mlsanity.com/paper/2505.22501/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)
- Synthetic data and self-improvement for LLMs / [Self-evolving search agents](https://mlsanity.com/synth/self-evolving-search-agents/)

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [Fine-tuning Language Models for Factuality](https://mlsanity.com/paper/2311.08401/) (2023)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Agents: An Open-source Framework for Autonomous Language Agents](https://mlsanity.com/paper/2309.07870/) (2023)
- [Post Hoc Explanations of Language Models Can Improve Language Models](https://mlsanity.com/paper/2305.11426/) (2023)
- [Plausible Extractive Rationalization through Semi-Supervised Entailment Signal](https://mlsanity.com/paper/2402.08479/) (2024)

## Built on by (most cited first, 30 of 43)

- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization](https://mlsanity.com/paper/2507.15061/) (2025)
- [Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL](https://mlsanity.com/paper/2508.13167/) (2025)
- [Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning](https://mlsanity.com/paper/2511.16043/) (2025)
- [Deep Research: A Survey of Autonomous Research Agents](https://mlsanity.com/paper/2508.12752/) (2025)
- [Dr. Zero: Self-Evolving Search Agents without Training Data](https://mlsanity.com/paper/2601.07055/) (2026)
- [Scaling Agents via Continual Pre-training](https://mlsanity.com/paper/2509.13310/) (2025)
- [Agentic Entropy-Balanced Policy Optimization](https://mlsanity.com/paper/2510.14545/) (2025)
- [Flash-Searcher: Fast and Effective Web Agents via DAG-Based Parallel Execution](https://mlsanity.com/paper/2509.25301/) (2025)
- [ExpSeek: Self-Triggered Experience Seeking for Web Agents](https://mlsanity.com/paper/2601.08605/) (2026)
- [Beyond Turn Limits: Training Deep Search Agents with Dynamic Context Window](https://mlsanity.com/paper/2510.08276/) (2025)
- [BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions](https://mlsanity.com/paper/2510.10666/) (2025)
- [Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain](https://mlsanity.com/paper/2603.02218/) (2026)
- [Reinforcement Learning Foundations for Deep Research Systems: A Survey](https://mlsanity.com/paper/2509.06733/) (2025)
- [ParallelMuse: Agentic Parallel Thinking for Deep Information Seeking](https://mlsanity.com/paper/2510.24698/) (2025)
- [Repurposing Synthetic Data for Fine-grained Search Agent Supervision](https://mlsanity.com/paper/2510.24694/) (2025)
- [Alita-G: Self-Evolving Generative Agent for Agent Generation](https://mlsanity.com/paper/2510.23601/) (2025)
- [LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services](https://mlsanity.com/paper/2512.07436/) (2025)
- [SkillMaster: Toward Autonomous Skill Mastery in LLM Agents](https://mlsanity.com/paper/2605.08693/) (2026)
- [WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection](https://mlsanity.com/paper/2510.18798/) (2025)
- [Deep Literature Survey Automation with an Iterative Workflow](https://mlsanity.com/paper/2510.21900/) (2025)
- [Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs](https://mlsanity.com/paper/2510.00861/) (2025)
- [A2FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning](https://mlsanity.com/paper/2510.12838/) (2025)
- [DLLM Agent: See Farther, Run Faster](https://mlsanity.com/paper/2602.07451/) (2026)
- [DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping](https://mlsanity.com/paper/2510.12979/) (2025)
- [Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization](https://mlsanity.com/paper/2602.22675/) (2026)
- [AutoAgent: Evolving Cognition and Elastic Memory Orchestration for Adaptive Agents](https://mlsanity.com/paper/2603.09716/) (2026)
- [Rethinking the Design of Reinforcement Learning-Based Deep Research Agents](https://mlsanity.com/paper/2510.15862/) (2025)
- [CubeBench: Diagnosing Interactive, Long-Horizon Spatial Reasoning Under Partial Observations](https://mlsanity.com/paper/2512.23328/) (2025)
- [ScholarGym: Benchmarking Large Language Model Capabilities in the Information-Gathering Stage of Deep Research](https://mlsanity.com/paper/2601.21654/) (2026)

## Abstract (arXiv)

The rapid advancement of large language models (LLMs) has transformed the landscape of agentic information seeking capabilities through the integration of tools such as search engines and web browsers. However, current mainstream approaches for enabling LLM web search proficiency face significant challenges: supervised fine-tuning struggles with data production in open-search domains, while RL converges quickly, limiting their data utilization efficiency. To address these issues, we propose EvolveSearch, a novel iterative self-evolution framework that combines SFT and RL to enhance agentic web search capabilities without any external human-annotated reasoning data. Extensive experiments on seven multi-hop question-answering (MHQA) benchmarks demonstrate that EvolveSearch consistently improves performance across iterations, ultimately achieving an average improvement of 4.7% over the current state-of-the-art across seven benchmarks, opening the door to self-evolution agentic capabilities in open web search domains.