mlsanityData to 14 Oct 2026

PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation

arXiv 2509.19128 ↗First posted 23 Sep 2025cs.LGcited by 32

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning (RL) is increasingly utilized to enhance the reasoning capabilities of Large Language Models (LLMs). However, effectively scaling these RL methods presents significant challenges, primarily due to the difficulty in maintaining high AI accelerator utilization without generating stale, off-policy data that harms common RL algorithms. This paper introduces PipelineRL, an approach designed to achieve a superior trade-off between hardware efficiency and data on-policyness for LLM training. PipelineRL employs concurrent asynchronous data generation and model training, distinguished by the novel in-flight weight updates. This mechanism allows the LLM generation engine to receive updated model weights with minimal interruption during the generation of token sequences, thereby maximizing both the accelerator utilization and the freshness of training data. Experiments conducted on long-form reasoning tasks using 128 H100 GPUs demonstrate that PipelineRL achieves approximately ~ 2x faster learning compared to conventional RL baselines while maintaining highly on-policy training data. A scalable and modular open-source implementation of PipelineRL is also released as a key contribution.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 30 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

20 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  2. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  3. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  10. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  11. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  12. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
2 more
  1. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  2. 2019Importance Resampling for Off-policy Predictioncited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 30.

32 papers.

Sort:

By area:

  1. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  2. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  3. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  4. 2026Composer 2 Technical Reportcited by 22
  5. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
  6. 2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoningcited by 14
  7. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 11
  8. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 9
  9. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 9
  10. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  11. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 5
  12. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 5
20 more
  1. 2026Not All LLM Reasoning is Visible in the Chain-of-Thoughtcited by 5
  2. 2026GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learningcited by 4
  3. 2026ZAYA1-8B Technical Reportcited by 4
  4. 2026FrogNano: Training a 4B Coding Agent via Online Task Synthesiscited by 4
  5. 2026QED-Nano: Teaching a Tiny Model to Prove Hard Theoremscited by 3
  6. 2026Score Centering Stabilizes Off-policy Reinforcement Learningcited by 3
  7. 2026Staleness-Learning Rate Scaling Laws for Asynchronous RLHFcited by 2
  8. 2026SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rolloutscited by 2
  9. 2026When Do Larger Batches Help Scale LLM Reinforcement Learning?cited by 2
  10. 2026CLaaS: Continual learning as a service for sample efficient online learningcited by 1
  11. 2026Learning to Solve Hard Problems in RL for LLMs by Never Giving Upcited by 1
  12. 2026KV-streams for Efficient Compaction in Agentic Reinforcement Learningcited by 1
  13. 2026Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient Reasoningcited by 0
  14. 2026Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learningcited by 0
  15. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RLcited by 0
  16. 2026An Open Recipe for IMO Gold: Training Nemotron for Olympiad Mathematicscited by 0
  17. 2026ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewardscited by 0
  18. 2026ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learningcited by 0
  19. 2026Target-ESS Tempered Policy Optimization (TETPO): Smooth Variance Control via Bridge Importance Samplingcited by 0
  20. 2026ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training
  2. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling
  3. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
  4. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
  5. 2025RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning
  6. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
  7. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  8. 2025TTRL: Test-Time Reinforcement Learning