# Search Self-play: Pushing the Frontier of Agent Capability without Supervision

arXiv: https://arxiv.org/abs/2510.18821 · HTML: https://mlsanity.com/paper/2510.18821/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 61 of its 83 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 30 of 61)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning](https://mlsanity.com/paper/2411.02337/) (2024)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [R-Zero: Self-Evolving Reasoning LLM from Zero Data](https://mlsanity.com/paper/2508.05004/) (2025)

## Built on by (most cited first, 30 of 34)

- [Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning](https://mlsanity.com/paper/2511.16043/) (2025)
- [VisPlay: Self-Evolving Vision-Language Models from Images](https://mlsanity.com/paper/2511.15661/) (2025)
- [Dr. Zero: Self-Evolving Search Agents without Training Data](https://mlsanity.com/paper/2601.07055/) (2026)
- [Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics](https://mlsanity.com/paper/2602.10885/) (2026)
- [From Context to Skills: Can Language Models Learn from Context Skillfully?](https://mlsanity.com/paper/2604.27660/) (2026)
- [Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills](https://mlsanity.com/paper/2607.22529/) (2026)
- [When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoning](https://mlsanity.com/paper/2602.10560/) (2026)
- [Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning](https://mlsanity.com/paper/2603.15611/) (2026)
- [π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data](https://mlsanity.com/paper/2604.14054/) (2026)
- [Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remember](https://mlsanity.com/paper/2607.29468/) (2026)
- [TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment](https://mlsanity.com/paper/2601.18292/) (2026)
- [CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR](https://mlsanity.com/paper/2603.10101/) (2026)
- [Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence](https://mlsanity.com/paper/2608.31075/) (2026)
- [Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL](https://mlsanity.com/paper/2605.22217/) (2026)
- [Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill](https://mlsanity.com/paper/2606.03980/) (2026)
- [GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization](https://mlsanity.com/paper/2606.16771/) (2026)
- [SearchMaster: Grounded and Regulated Self-Play for Search Agents](https://mlsanity.com/paper/2608.01822/) (2026)
- [Knowledge-Graph Paths as Intermediate Supervision for Self-Evolving Search Agents](https://mlsanity.com/paper/2605.05702/) (2026)
- [Superintelligent Retrieval Agent: The Next Frontier of Agentic Retrieval](https://mlsanity.com/paper/2605.06647/) (2026)
- [Self-Evolving Deep Research via Joint Generation and Evaluation](https://mlsanity.com/paper/2606.04507/) (2026)
- [EXG: Self-Evolving Agents with Experience Graphs](https://mlsanity.com/paper/2605.17721/) (2026)
- [Delegation Intelligence in Deep Search: A Controllable Framework for Disentangled Capability Diagnosis](https://mlsanity.com/paper/2607.23524/) (2026)
- [Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Design](https://mlsanity.com/paper/2608.10299/) (2026)
- [False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents](https://mlsanity.com/paper/2609.39102/) (2026)
- [MedZERO: Self-Evolving Agents for Open-Ended Medical Reasoning Through Controlled Knowledge Accumulation](https://mlsanity.com/paper/2610.08327/) (2026)
- [A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions](https://mlsanity.com/paper/2604.17312/) (2026)
- [Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models](https://mlsanity.com/paper/2604.20148/) (2026)
- [CAPF: Guiding Search-Agent Rollouts with Credit-Attenuated Privileged Feedback](https://mlsanity.com/paper/2606.01830/) (2026)
- [Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teacher](https://mlsanity.com/paper/2606.25407/) (2026)
- [ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning](https://mlsanity.com/paper/2609.01058/) (2026)

## Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become the mainstream technique for training LLM agents. However, RLVR highly depends on well-crafted task queries and corresponding ground-truth answers to provide accurate rewards, which requires significant human effort and hinders the scaling of RL processes, especially in agentic scenarios. Although a few recent works explore task synthesis methods, the difficulty of generated agentic tasks can hardly be controlled to provide effective RL training advantages. To achieve agentic RLVR with higher scalability, we explore self-play training for deep search agents, in which the learning LLM utilizes multi-turn search engine calling and acts simultaneously as both a task proposer and a problem solver. The task proposer aims to generate deep search queries with well-defined ground-truth answers and increasing task difficulty. The problem solver tries to handle the generated search queries and output the correct answer predictions. To ensure that each generated search query has accurate ground truth, we collect all the searching results from the proposer's trajectory as external knowledge, then conduct retrieval-augmentation generation (RAG) to test whether the proposed query can be correctly answered with all necessary search documents provided. In this search self-play (SSP) game, the proposer and the solver co-evolve their agent capabilities through both competition and cooperation. With substantial experimental results, we find that SSP can significantly improve search agents' performance uniformly on various benchmarks without any supervision under both from-scratch and continuous RL training setups. The code is at https://github.com/Qwen-Applications/SSP.