mlsanityData to 14 Oct 2026

Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards

arXiv 2610.15888 ↗First posted 11 Oct 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distribution itself: under a fixed rollout budget, which distribution over states minimizes the variance of the policy-gradient estimator? The answer admits a closed-form Neyman allocation, μvarπ(s) propto dπ(s),σπ(s), which connects naturally to GRPO's group-relative advantage and induces an emergent reverse curriculum as the policy improves, with no externally imposed schedule. Building on this principle, we propose VSR (Variance-driven State Resampling), a plug-in modification to GRPO using a prefix replay buffer and a zero-cost variance proxy. On mathematical reasoning benchmarks and long-horizon agentic tasks (Sokoban, BabyAI), VSR outperforms GRPO and GiGPO on most benchmarks with fewer generated tokens per update.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 43 references that are in the map. First posted on arXiv 11 Oct 2026.

Builds on

38 of 43 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Self-Hinting Language Models Enhance Reinforcement Learning3 Feb
  2. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration26 Jan
  3. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives6 Oct
  4. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning20 Jun
  5. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  6. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  7. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025Process Reinforcement through Implicit Rewards3 Feb
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb

Other references · 27

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  8. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  9. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  10. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  11. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  12. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
15 more
  1. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 150
  2. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 100
  3. 2022Language Models Can Teach Themselves to Program Bettercited by 78
  4. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  5. 2025Self-Questioning Language Modelscited by 50
  6. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 50
  7. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  8. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 34
  9. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 27
  10. 2025LADDER: Self-Improving LLMs Through Recursive Problem Decompositioncited by 25
  11. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 24
  12. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 24
  13. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 20
  14. 2026PivotRL: High Accuracy Agentic Post-Training at Low Compute Costcited by 14
  15. 2025Propose, Solve, Verify: Self-Play Through Formal Verificationcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
  2. 2025Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewards
  3. 2025Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
  4. 2026Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States
  5. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards
  6. 2026Verifier-Induced Support Reshaping in On-Policy Optimization
  7. 2026Variance Reduction Based Experience Replay for Policy Optimization
  8. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning