mlsanityData to 7 Oct 2026

Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?

arXiv 2510.01161 ↗First posted 1 Oct 2025cs.LG cs.AIcited by 49

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning has been central to recent advances in large language model reasoning, but most algorithms rely on on-policy training that demands fresh rollouts at every update, limiting efficiency and scalability. Asynchronous RL systems alleviate this by decoupling rollout generation from training, yet their effectiveness hinges on tolerating large staleness in rollout data, a setting where existing methods either degrade in performance or collapse. We revisit this challenge and uncover a prosperity-before-collapse phenomenon: stale data can be as informative as on-policy data if exploited properly. Building on this insight, we introduce M2PO (Second-Moment Trust Policy Optimization), which constrains the second moment of importance weights to suppress only extreme outliers while preserving informative updates. Notably, M2PO sharply reduces the fraction of clipped tokens under high staleness (from 1.22% to 0.06% over training), precisely masking high-variance tokens while maintaining stable optimization. Extensive evaluation across six models (from 1.7B to 32B) and eight benchmarks shows that M2PO delivers stable off-policy training even with data stale by at least 256 model updates and matches on-policy performance.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 44 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

36 of 44 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL26 Aug
  2. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization11 Aug
  3. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  7. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  8. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  9. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
4 more
  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  2. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct
  3. 2023Let's Verify Step by Step31 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  12. 2025Group Sequence Policy Optimizationcited by 788
8 more
  1. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  2. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  3. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  4. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  5. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  6. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  7. 2025One-shot Entropy Minimizationcited by 32
  8. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 48.

49 papers.

Sort:

By area:

  1. 2026R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplificationcited by 12
  2. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillationcited by 12
  3. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  4. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 10
  5. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  6. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 9
  7. 2026Efficient RL Training for LLMs with Experience Replaycited by 8
  8. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimizationcited by 7
  9. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  10. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 7
  11. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learningcited by 7
  12. 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 5
37 more
  1. 2026Rethinking the Divergence Regularization in LLM RLcited by 5
  2. 2026Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learningcited by 4
  3. 2026GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Controlcited by 4
  4. 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correctioncited by 4
  5. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 4
  6. 2026Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Trainingcited by 4
  7. 2026The Art of Efficient Reasoning: Data, Reward, and Optimizationcited by 3
  8. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learningcited by 3
  9. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimizationcited by 3
  10. 2026Learning Agentic Policy from Action Guidancecited by 3
  11. 2026AsyncOPD: How Stale Can On-Policy Distillation Be?cited by 3
  12. 2026ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learningcited by 2
  13. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 2
  14. 2026Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMscited by 1
  15. 2026RLHFless: Serverless Computing for Efficient RLHFcited by 1
  16. 2026Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Modelscited by 1
  17. 2026AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMscited by 1
  18. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 1
  19. 2026Off-Policy Value-Based Reinforcement Learning for Large Language Modelscited by 0
  20. 2026Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient Reasoningcited by 0
  21. 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVRcited by 0
  22. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 0
  23. 2026When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVRcited by 0
  24. 2026PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVRcited by 0
  25. 2026Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHFcited by 0
  26. 2026Staleness-Learning Rate Scaling Laws for Asynchronous RLHFcited by 0
  27. 2026UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemmacited by 0
  28. 2026Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learningcited by 0
  29. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RLcited by 0
  30. 2026Performance, Efficiency and Collapse -- Advantages and Challenges in Offline Post-training of Code LLMscited by 0
  31. 2026MInTRL: Off-policy Intervention can boost On-policy RLcited by 0
  32. 2026Towards Full Pipeline FP8 Reinforcement Learning for LLMscited by 0
  33. 2026WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learningcited by 0
  34. 2026Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Trainingcited by 0
  35. 2026Cool the Sampler, Not the Learner: Sampling Temperature Moves the Staleness Cliff of Importance-Corrected GRPOcited by 0
  36. 2026REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reusecited by 0
  37. 2026Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
  2. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
  3. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning
  4. 2026RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
  5. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation
  6. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
  7. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model
  8. 2026COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning