mlsanityData to 14 Oct 2026

Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR

arXiv 2508.14029 ↗First posted 19 Aug 2025cs.CLcited by 54

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a key paradigm for post-training Large Language Models (LLMs), particularly for complex reasoning tasks. However, vanilla RLVR training has been shown to improve Pass@1 performance at the expense of policy entropy, leading to reduced generation diversity and limiting the Pass@k performance, which typically represents the upper bound of LLM reasoning capability. In this paper, we systematically analyze the policy's generation diversity from the perspective of training problems and find that augmenting and updating training problems helps mitigate entropy collapse during training. Based on these observations, we propose an online Self-play with Variational problem Synthesis (SvS) strategy for RLVR training, which uses the policy's correct solutions to synthesize variational problems while ensuring their reference answers remain identical to the originals. This self-improving strategy effectively maintains policy entropy during training and substantially improves Pass@k compared with standard RLVR, sustaining prolonged improvements and achieving absolute gains of 18.3% and 22.8% in Pass@32 performance on the competition-level AIME24 and AIME25 benchmarks, as well as on code generation tasks. Experiments on 12 reasoning benchmarks across varying model sizes from 3B to 32B consistently demonstrate the generalizability and robustness of SvS.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 68 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

55 of 68 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  3. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning10 Jun
  4. 2025TreeRPO: Tree Relative Policy Optimization5 Jun
  5. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  6. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  8. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning22 May
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  10. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  11. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  12. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
6 more
  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2023Let's Verify Step by Step31 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
25 more
  1. 2025Group Sequence Policy Optimizationcited by 835
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  4. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  5. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  6. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  7. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  8. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 186
  9. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  10. 2016Dual Learning for Machine Translationcited by 148
  11. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  12. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  13. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  14. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70
  15. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  16. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Spacecited by 43
  17. 2025One-shot Entropy Minimizationcited by 32
  18. 2025Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspectivecited by 29
  19. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  20. 2025PromptCoT: Synthesizing Olympiad-level Problems for Mathematical Reasoning in Large Language Modelscited by 22
  21. 2024Explore the Reasoning Capability of LLMs in the Chess Testbedcited by 19
  22. 2025TL;DR: Too Long, Do Re-weighting for Efficient LLM Reasoning Compressioncited by 16
  23. 2025Synthetic Data RL: Task Definition Is All You Needcited by 15
  24. 2024Task Oriented In-Domain Data Augmentationcited by 6
  25. 2025GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectorscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 29.

54 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
  4. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  5. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 43
  6. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  7. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
  8. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  9. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 34
  10. 2025Aegis: Automated Error Generation and Attribution for Multi-Agent Systemscited by 22
  11. 2026Composer 2 Technical Reportcited by 22
  12. 2026Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulationcited by 19
38 more
  1. 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
  2. 2025LoopTool: Closing the Data-Training Loop for Robust LLM Tool Callscited by 17
  3. 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
  4. 2025Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillationcited by 15
  5. 2026EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RLcited by 14
  6. 2025CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoningcited by 13
  7. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  8. 2025Dynamic Generation of Multi-LLM Agents Communication Topologies with Graph Diffusion Modelscited by 11
  9. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  10. 2025Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learningcited by 10
  11. 2025Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scalingcited by 9
  12. 2025Towards Understanding Self-play for LLM Reasoningcited by 8
  13. 2026Mnemis: Dual-Route Retrieval on Hierarchical Graphs for Long-Term LLM Memorycited by 8
  14. 2026When Silence Is Golden: Can LLMs Learn to Abstain in Temporal QA and Beyond?cited by 7
  15. 2026OmniOPD: Logit-Free On-Policy Distillation via Speculative Verificationcited by 7
  16. 2025Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Modelscited by 6
  17. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewardscited by 6
  18. 2026Efficient RLVR Training via Weighted Mutual Information Data Selectioncited by 6
  19. 2026Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectificationcited by 5
  20. 2026From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoningcited by 5
  21. 2026Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalabilitycited by 4
  22. 2026Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RLcited by 4
  23. 2026CoEvoKG: Co-Evolving Knowledge Graphs with Self-Evolving Search Agentscited by 3
  24. 2025RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initializationcited by 2
  25. 2025Decouple to Generalize: Context-First Self-Evolving Learning for Data-Scarce Vision-Language Reasoningcited by 2
  26. 2026Preventing Curriculum Collapse in Self-Evolving Reasoning Systemscited by 2
  27. 2026Policy Improvement Reinforcement Learningcited by 2
  28. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVRcited by 2
  29. 2025Gold-Medal-Level Olympiad Geometry Solving with Efficient Heuristic Auxiliary Constructionscited by 1
  30. 2026PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoningcited by 1
  31. 2026Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approachcited by 1
  32. 2026MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Modelscited by 1
  33. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1
  34. 2026Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMscited by 1
  35. 2026From Solvers to Research: Large Language Model-Driven Formal Mathematics at the Research Frontiercited by 1
  36. 2026Learning What to Practice: Diagnosis-Guided Self-Evolution for Language Modelscited by 1
  37. 2025AdaR: A Framework for Equipping LLMs with Adaptive Reasoningcited by 0
  38. 2026CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computationcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  2. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
  3. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
  4. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  5. 2026Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning
  6. 2026Linear Dynamics in the RLVR Training of Large Language Models
  7. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  8. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective