mlsanityData to 14 Oct 2026

SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling

arXiv 2603.23414 ↗First posted 24 Mar 2026cs.LG cs.AIcited by 6

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Scaling reinforcement learning (RL) has shown strong promise for enhancing the reasoning abilities of large language models (LLMs), particularly in tasks requiring long chain-of-thought generation. However, RL training efficiency is often bottlenecked by the rollout phase, which can account for up to 70% of total training time when generating long trajectories (e.g., 16k tokens), due to slow autoregressive generation and synchronization overhead between rollout and policy updates. We propose SortedRL, an online length-aware scheduling strategy designed to address this bottleneck by improving rollout efficiency and maintaining training stability. SortedRL reorders rollout samples based on output lengths, prioritizing short samples forming groups for early updates. This enables large rollout batches, flexible update batches, and near on-policy micro-curriculum construction simultaneously. To further accelerate the pipeline, SortedRL incorporates a mechanism to control the degree of off-policy training through a cache-based mechanism, and is supported by a dedicated RL infrastructure that manages rollout and update via a stateful controller and rollout buffer. Experiments using LLaMA-3.1-8B and Qwen-2.5-32B on diverse tasks, including logical puzzles, and math challenges like AIME 24, Math 500, and Minerval, show that SortedRL reduces RL training bubble ratios by over 50%, while attaining 3.9% to 18.4% superior performance over baseline given same amount of data.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 37 references that are in the map. First posted on arXiv 24 Mar 2026.

Builds on

25 of 37 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
7 more
  1. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  2. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  3. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  4. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  5. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  6. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  7. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  2. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 6
  3. 2026DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Trainingcited by 2
  4. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  5. 2026S3R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decodingcited by 0
  6. 2026Efficient Resource-Shiftable RL Post-training with Generative Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
  2. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  3. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
  4. 2025FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models
  5. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  6. 2026Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning
  7. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
  8. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training