mlsanityData to 14 Oct 2026

RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training

arXiv 2509.21009 ↗First posted 25 Sep 2025cs.DC cs.LGcited by 35

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning (RL) is a pivotal post-training technique for enhancing the reasoning capabilities of Large Language Models (LLMs). However, synchronous RL post-training often suffers from significant GPU underutilization, referred to as bubbles, caused by imbalanced response lengths within rollout steps. Many RL systems attempt to alleviate this problem by relaxing synchronization, but this can compromise training accuracy. In this paper, we introduce tail batching, a novel rollout scheduling strategy for synchronous RL that systematically consolidates prompts leading to long-tail responses into a small subset of rollout steps (long rounds), while ensuring that the majority of steps (short rounds) involve only balanced, short rollouts. By excluding long responses from short rounds and rescheduling them into a few designated long rounds, tail batching effectively reduces GPU idle time during rollouts and significantly accelerates RL training without sacrificing accuracy. We present RollPacker, a system that fully harnesses the benefits of tail batching through holistic optimizations across all three RL stages: elastic parallelism adaptation for rollout, dynamic resource allocation and scheduling for reward, and stream-based training. Empirical results show that RollPacker achieves a 2.03x-2.56x end-to-end training time reduction compared to veRL and up to 2.24x speedup compared to RLHFuse for the Qwen2.5 family of LLMs on up to 128 H800 GPUs.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 48 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

29 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL26 Aug
  2. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
  3. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning10 Jun
  4. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library6 Jun
  5. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  6. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  7. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 21

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  11. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  12. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
9 more
  1. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
  2. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  3. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 119
  4. 2017Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learningcited by 40
  5. 2021Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learningcited by 34
  6. 2024Fast State Restoration in LLM Serving with HCachecited by 33
  7. 2021Maximizing Parallelism in Distributed Training for Huge Neural Networkscited by 23
  8. 2021Tesseract: Parallelize the Tensor Parallelism Efficientlycited by 19
  9. 2025DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Trainingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 27.

35 papers.

Sort:

By area:

  1. 2026MiMo-V2-Flash Technical Reportcited by 191
  2. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 30
  3. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  4. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
  5. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillationcited by 18
  6. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  7. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchronycited by 15
  8. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Draftercited by 14
  9. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 14
  10. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  11. 2025Klear-AgentForge: Forging Agentic Intelligence through Posttraining Scalingcited by 9
  12. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimizationcited by 7
23 more
  1. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  2. 2026TensorHub: Scalable and Elastic Weight Transfer for LLM RL Trainingcited by 4
  3. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
  4. 2026Not all tokens are needed(NAT): token efficient reinforcement learningcited by 3
  5. 2026AsyncOPD: How Stale Can On-Policy Distillation Be?cited by 3
  6. 2026WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learningcited by 3
  7. 2026Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learningcited by 2
  8. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learningcited by 2
  9. 2026Faster Synchronous On-Policy RL via Straggler-Aware Group Sizingcited by 2
  10. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 1
  11. 2026The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learningcited by 1
  12. 2026ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learningcited by 0
  13. 2026Off-Policy Value-Based Reinforcement Learning for Large Language Modelscited by 0
  14. 2026AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Servingcited by 0
  15. 2026ReCrit: Transition-Aware Reinforcement Learning for Scientific Critic Reasoningcited by 0
  16. 2026Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Trainingcited by 0
  17. 2026Belayer: Efficient Fault Tolerance for LLM Agentic RL Trainingcited by 0
  18. 2026WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learningcited by 0
  19. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  20. 2026ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learningcited by 0
  21. 2026RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learningcited by 0
  22. 2026Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learningcited by 0
  23. 2026Efficient Resource-Shiftable RL Post-training with Generative Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling
  2. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation
  3. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  4. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
  5. 2025Laminar: A Scalable Asynchronous RL Post-Training Framework
  6. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  7. 2026Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR
  8. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning