mlsanityData to 7 Oct 2026

How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization

arXiv 2602.19208 ↗First posted 22 Feb 2026cs.LG cs.AIcited by 14

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for Large Language Model (LLM) reasoning, yet current methods face key challenges in resource allocation and policy optimization dynamics: (i) uniform rollout allocation ignores gradient variance heterogeneity across problems, and (ii) the softmax policy structure causes gradient attenuation for high-confidence correct actions, while excessive gradient updates may destabilize training. Therefore, we propose DynaMO, a theoretically-grounded dual-pronged optimization framework. At the sequence level, we prove that uniform allocation is suboptimal and derive variance-minimizing allocation from the first principle, establishing Bernoulli variance as a computable proxy for gradient informativeness. At the token level, we develop gradient-aware advantage modulation grounded in theoretical analysis of gradient magnitude bounds. Our framework compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. Extensive experiments conducted on a diverse range of mathematical reasoning benchmarks demonstrate consistent improvements over strong RLVR baselines. Our implementation is available at: https://github.com/GithubX-F/DynaMO-RL.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 36 references that are in the map. First posted on arXiv 22 Feb 2026.

Builds on

20 of 36 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  9. 2025Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Trainingcited by 42
  10. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 33
  11. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 23
  12. 2026Verified Critical Step Optimization for LLM Agentscited by 13
1 more
  1. 2025Logit Dynamics in Softmax Policy Gradient Methodscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Taskscited by 7
  2. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinementcited by 6
  3. 2026Learning from Contrasts: Synthesizing Reasoning Paths from Diverse Search Trajectoriescited by 4
  4. 2026Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderationcited by 3
  5. 2026From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Spacecited by 3
  6. 2026DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewardscited by 3
  7. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVRcited by 2
  8. 2026LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Modelscited by 2
  9. 2026Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learningcited by 2
  10. 2026Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generationcited by 1
  11. 2026How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learningcited by 1
  12. 2026Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learningcited by 0
2 more
  1. 2026Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regimecited by 0
  2. 2026SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
  2. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
  3. 2026Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients
  4. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
  5. 2026What are Key Factors for Updates in RL for LLM Reasoning?
  6. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
  7. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  8. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement