mlsanityData to 14 Oct 2026

SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data

arXiv 2505.20347 ↗First posted 25 May 2025cs.CL cs.AIcited by 51

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advances have demonstrated the effectiveness of Reinforcement Learning (RL) in improving the reasoning capabilities of Large Language Models (LLMs). However, existing works inevitably rely on high-quality instructions and verifiable rewards for effective training, both of which are often difficult to obtain in specialized domains. In this paper, we propose Self-play Reinforcement Learning (SeRL) to bootstrap LLM training with limited initial data. Specifically, SeRL comprises two complementary modules: self-instruction and self-rewarding. The former module generates additional instructions based on the available data at each training step, employing robust online filtering strategies to ensure instruction quality, diversity, and difficulty. The latter module introduces a simple yet effective majority-voting mechanism to estimate response rewards for additional instructions, eliminating the need for external annotations. Finally, SeRL performs conventional RL based on the generated data, facilitating iterative self-play learning. Extensive experiments on various reasoning benchmarks and across different LLM backbones demonstrate that the proposed SeRL yields results superior to its counterparts and achieves performance on par with those obtained by high-quality data with verifiable rewards. Our code is available at https://github.com/wantbook-book/SeRL.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 69 of its 93 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

69 of 93 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  2. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  5. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use7 Apr
  6. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
  7. 2025Reasoning with Reinforced Functional Token Tuning19 Feb
  8. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  9. 2023Let's Verify Step by Step31 May
  10. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 58

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  9. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
38 more
  1. 2024Self-Rewarding Language Modelscited by 681
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  4. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  6. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  7. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  8. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  9. 2024Iterative Reasoning Preference Optimizationcited by 246
  10. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  11. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  12. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  13. 2018Constituency Parsing with a Self-Attentive Encodercited by 199
  14. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  15. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  16. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  17. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  18. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 153
  19. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  20. 2018Multilingual Constituency Parsing with Self-Attention and Pre-Trainingcited by 117
  21. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
  22. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  23. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
  24. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 84
  25. 2025Self-rewarding correction for mathematical reasoningcited by 70
  26. 2024Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Comparecited by 66
  27. 2023Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculumcited by 65
  28. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  29. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  30. 2024WIPI: A New Web Threat for LLM-Driven Web Agentscited by 51
  31. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 39
  32. 2025A Survey of Direct Preference Optimizationcited by 36
  33. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  34. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  35. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samplescited by 25
  36. 20242AFC Prompting of Large Multimodal Models for Image Quality Assessmentcited by 20
  37. 2024Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labelscited by 19
  38. 2024Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaborationcited by 19

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 31.

51 papers.

Sort:

By area:

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 75
  4. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  5. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolutioncited by 60
  6. 2025Self-Questioning Language Modelscited by 50
  7. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 46
  8. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  9. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
  10. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 34
  11. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 34
  12. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 26
38 more
  1. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 23
  2. 2025FrontierCS: Evolving Challenges for Evolving Intelligencecited by 23
  3. 2026TTCS: Test-Time Curriculum Synthesis for Self-Evolvingcited by 19
  4. 2026G-Zero: Self-Play for Open-Ended Generation from Zero Datacited by 17
  5. 2026On-Policy Self-Distillation without Any Supervisioncited by 17
  6. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  7. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  8. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  9. 2025Better LLM Reasoning via Dual-Playcited by 8
  10. 2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 8
  11. 2026Active Zero: Self-Evolving Vision-Language Models through Active Environment Explorationcited by 8
  12. 2025A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Modelscited by 7
  13. 2026Autonomous Continual Learning for Environment Adaptation of Computer-Use Agentscited by 6
  14. 2025KnowRL: Teaching Language Models to Know What They Knowcited by 5
  15. 2026From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoningcited by 5
  16. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  17. 2026RISE: Recursive Improvement via Self-Extrapolating Policy Distillationcited by 5
  18. 2025Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compressioncited by 4
  19. 2026Adaptive Time Series Reasoning via Segment Selectioncited by 4
  20. 2025Tree of Preferences for Diversified Recommendationcited by 2
  21. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifierscited by 2
  22. 2026TTSR: Test-Time Self-Evolving via Reflectioncited by 2
  23. 2026Self-Evolving Deep Research via Joint Generation and Evaluationcited by 2
  24. 2025Meta-Learning Reinforcement Learning for Crypto-Return Predictioncited by 1
  25. 2026SPA-Cache: Singular Proxies for Adaptive Caching in Diffusion Language Modelscited by 1
  26. 2026TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesiscited by 1
  27. 2026GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modelingcited by 1
  28. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvementcited by 1
  29. 2025Zero-shot 3D Map Generation with LLM Agents: A Dual-Agent Architecture for Procedural Content Generationcited by 0
  30. 2026Sci-CoE: Co-evolving Scientific Reasoning LLMs via Geometric Consensus with Sparse Supervisioncited by 0
  31. 2026Dynamic Adversarial Reinforcement Learning for Robust Multimodal Large Language Modelscited by 0
  32. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  33. 2026Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learningcited by 0
  34. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  35. 2026CLORE: Content-Level Optimization for Reasoning Efficiencycited by 0
  36. 2026TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecastingcited by 0
  37. 2026Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teachercited by 0
  38. 2026Sharpen Before You Adapt: Data-Free Entry-State Sharpening for Test-Time Reinforcement Learningcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SERL: Self-Examining Reinforcement Learning on Open-Domain
  2. 2025Language Self-Play For Data-Free Training
  3. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
  4. 2025SSRL: Self-Search Reinforcement Learning
  5. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback
  6. 2026WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement
  7. 2025Towards Understanding Self-play for LLM Reasoning
  8. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models