mlsanityData to 7 Oct 2026

APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation

arXiv 2509.18521 ↗First posted 23 Sep 2025cs.LG cs.AIcited by 27

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) has become a cornerstone in advancing large-scale pre-trained language models (LLMs). Successive generations, including GPT-o series, DeepSeek-R1, Kimi-K1.5, Grok 4, and GLM-4.5, have relied on large-scale RL training to enhance reasoning and coding capabilities. To meet the community's growing RL needs, numerous RL frameworks have been proposed. However, RL training remains computationally expensive, with rollout generation accounting for more than 90% of total runtime. In addition, its efficiency is often constrained by the long-tail distribution of rollout response lengths, where a few lengthy responses stall entire batches, leaving GPUs idle and underutilized. As model and rollout sizes continue to grow, this bottleneck increasingly limits scalability. To address this challenge, we propose Active Partial Rollouts in Reinforcement Learning (APRIL), which mitigates long-tail inefficiency. In the rollout phase, APRIL over-provisions rollout requests, terminates once the target number of responses is reached, and recycles incomplete responses for continuation in future steps. This strategy ensures that no rollouts are discarded while substantially reducing GPU idle time. Experiments show that APRIL improves rollout throughput by 22.5% on average (at most 44%) across commonly used RL algorithms (GRPO, DAPO, GSPO), accelerates convergence, and achieves 2.1% on average(at most 8%) higher final accuracy across tasks. Moreover, APRIL is both framework and hardware agnostic, already integrated into the slime RL framework, and deployable on NVIDIA and AMD GPUs alike. Taken together, this work unifies system-level and algorithmic considerations in proposing APRIL, with the aim of advancing RL training efficiency and inspiring further optimizations in RL systems. Our codebase is available at https://github.com/RLsys-Foundation/APRIL

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

24 of 35 references.

References · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
12 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  2. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  3. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  4. 2025Learning to Reason under Off-Policy Guidancecited by 266
  5. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  6. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 108
  7. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  8. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 45
  9. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  10. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  11. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 37
  12. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 24.

27 papers.

Sort:

By area:

  1. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 43
  2. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  3. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Draftercited by 12
  4. 2026ERNIE 5.0 Technical Reportcited by 9
  5. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 9
  6. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  7. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 5
  8. 2026TensorHub: Scalable and Elastic Weight Transfer for LLM RL Trainingcited by 4
  9. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 3
  10. 2026Not all tokens are needed(NAT): token efficient reinforcement learningcited by 2
  11. 2026Learning to Reason with Curriculum I: Provable Benefits of Autocurriculumcited by 2
  12. 2026DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learningcited by 2
15 more
  1. 2026AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agentscited by 2
  2. 2026DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Trainingcited by 2
  3. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Schedulingcited by 2
  4. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  5. 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Searchcited by 1
  6. 2026Rollplex: Cross-Phase GPU Spatial Sharing for Vision Language Model Post-Trainingcited by 1
  7. 2026Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Modelscited by 0
  8. 2026Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspectivecited by 0
  9. 2026The Hitchhiker's Guide to Agentic AI: From Foundations to Systemscited by 0
  10. 2026Instella-MoE Technical Reportcited by 0
  11. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  12. 2026Allspark: Weak to Strong Transfer via Alternating Chain of Thoughtcited by 0
  13. 2026Nereus: Adaptive Parallelism for LLM Post-Trainingcited by 0
  14. 2026DivOPD: Spread Wide, Look Close for Asynchronous On-Policy Distillation of Multi-turn Agentscited by 0
  15. 2026Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
  2. 2026RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
  3. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
  4. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
  5. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
  6. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  7. 2026Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR
  8. 2026Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning