mlsanityData to 7 Oct 2026

SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning

arXiv 2609.36552 ↗First posted 29 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Maximum Likelihood Reinforcement Learning (MaxRL) targets prompt-wise log-success and has shown strong performance on reasoning tasks. Under finite rollout budgets, however, the estimator used by MaxRL attenuates each prompt's likelihood gradient by a factor that depends on its success probability and rollout count. Under uniform rollout allocation, the common rollout count fails to compensate for success-dependent attenuation, leaving low-success prompts more strongly attenuated and distorting their relative contributions to the expected aggregate gradient. We introduce SERA (Scale-Equalized Rollout Allocation), which redistributes a fixed rollout budget to approximately equalize these finite-rollout scaling factors. Building on our theoretical analysis of how finite rollouts distort prompt-wise likelihood gradients, we formulate the allocation as a fixed-budget max--min problem, derive a waterline solution to its continuous relaxation, and introduce a multiplicity correction to remove the additional prompt weighting induced by heterogeneous rollout counts. Experiments show stronger alignment with exact likelihood gradients in a controlled ImageNet setting and improved multi-sample solution coverage over MaxRL on maze navigation and mathematical reasoning under matched training rollout budgets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 53 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

47 of 53 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
35 more
  1. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  3. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  5. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  6. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  7. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  8. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  9. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  10. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 52
  11. 2025Do Large Language Model Benchmarks Test Reliability?cited by 50
  12. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 49
  13. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  14. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Explorationcited by 40
  15. 2026Maximum Likelihood Reinforcement Learningcited by 29
  16. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 23
  17. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMscited by 23
  18. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 23
  19. 2025What is the objective of reasoning with reinforcement learning?cited by 16
  20. 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
  21. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 15
  22. 2026How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimizationcited by 14
  23. 2025Pass@k Metric for RLVR: A Diagnostic Tool of Exploration, But Not an Objectivecited by 11
  24. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 9
  25. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  26. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learningcited by 7
  27. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 6
  28. 2026Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversitycited by 6
  29. 2026Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVRcited by 5
  30. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  31. 2026Cross-Epoch Adaptive Rollout Optimization for RL Post-Trainingcited by 2
  32. 2026Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVRcited by 2
  33. 2026PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVRcited by 1
  34. 2026Efficient RLVR Scheduling via Graph-Structured Online Difficulty Estimationcited by 1
  35. 2026Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVRcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning
  2. 2026CERO: Where and When to Allocate Rollouts for RL Post-Training
  3. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
  4. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library
  5. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
  6. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
  7. 2026On Training in Imagination
  8. 2026EchoRL: Reinforcement Learning via Rollout Echoing