mlsanityData to 14 Oct 2026

Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library

arXiv 2506.06122 ↗First posted 6 Jun 2025cs.LG cs.DCcited by 101

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

We introduce ROLL, an efficient, scalable, and user-friendly library designed for Reinforcement Learning Optimization for Large-scale Learning. ROLL caters to three primary user groups: tech pioneers aiming for cost-effective, fault-tolerant large-scale training, developers requiring flexible control over training workflows, and researchers seeking agile experimentation. ROLL is built upon several key modules to serve these user groups effectively. First, a single-controller architecture combined with an abstraction of the parallel worker simplifies the development of the training pipeline. Second, the parallel strategy and data transfer modules enable efficient and scalable training. Third, the rollout scheduler offers fine-grained management of each sample's lifecycle during the rollout stage. Fourth, the environment worker and reward worker support rapid and flexible experimentation with agentic RL algorithms and reward designs. Finally, AutoDeviceMapping allows users to assign resources to different models flexibly across various stages.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 52 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

37 of 52 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  2. 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning15 Apr
  3. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  4. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  5. 2024OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning22 Dec
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  11. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 848
18 more
  1. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  4. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  6. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  7. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  8. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  9. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  10. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
  11. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 119
  12. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 88
  13. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  14. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  15. 2023DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Trainingcited by 29
  16. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  17. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 8
  18. 20242D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervisioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 43; 2026: 58.

101 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 109
  5. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  6. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  7. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 56
  8. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
  9. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  10. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  11. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 31
  12. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 30
38 more
  1. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 30
  2. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 26
  3. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 25
  4. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  5. 2025GEM: A Gym for Agentic LLMscited by 23
  6. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  7. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  8. 2026SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Trainingcited by 22
  9. 2025ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimizationcited by 20
  10. 2025Logics-Parsing Technical Reportcited by 17
  11. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  12. 2025CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalizationcited by 16
  13. 2025MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMscited by 16
  14. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 16
  15. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 15
  16. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchronycited by 15
  17. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  18. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimizationcited by 14
  19. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 14
  20. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 14
  21. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learningcited by 13
  22. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learningcited by 13
  23. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximationcited by 12
  24. 2025RecGPT Technical Reportcited by 11
  25. 2026PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalizationcited by 11
  26. 2026Verifiable Process Rewards for Agentic Reasoningcited by 11
  27. 2025TaoSR1: The Thinking Model for E-commerce Relevance Searchcited by 10
  28. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 10
  29. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 10
  30. 2026Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Samplingcited by 9
  31. 2025Identity-Preserving Image-to-Video Generation via Reward-Guided Optimizationcited by 8
  32. 2026Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scalecited by 7
  33. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 6
  34. 2025TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilotscited by 5
  35. 2026GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Controlcited by 5
  36. 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correctioncited by 5
  37. 2025TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevancecited by 4
  38. 2025Retrieval-GRPO: A Multi-Objective Reinforcement Learning Framework for Dense Retrieval in Taobao Searchcited by 4

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
  2. 2026WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learning
  3. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions
  4. 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration
  5. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
  6. 2026Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits
  7. 2026RL2ML: Finite-Rollout Surrogate Objectives from Reinforcement Learning to Maximum Likelihood
  8. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training