mlsanityData to 14 Oct 2026

Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding

arXiv 2604.26779 ↗First posted 29 Apr 2026cs.LG cs.CLcited by 9

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

RL post-training of frontier language models is increasingly bottlenecked by autoregressive rollout generation, making rollout acceleration a central systems challenge. Many existing efficiency methods improve throughput by changing the rollout or optimization regime, for example, through off-policy execution, replay, or lower-precision generation. We study speculative decoding as a lossless acceleration primitive for RL rollouts that preserves the target model's output distribution. We implement speculative decoding in NeMo-RL with a vLLM backend, supporting both synchronous and asynchronous pipelines and enabling speculation during RL rollouts. This benefit is realizable across speculation mechanisms, such as pretrained MTP heads, small external draft models or even techniques such as Eagle3, which are traditionally applied after RL phase. This yields a deployment path for state-of-the-art speculative decoding inside RL training. In a reasoning post-training workload at 8B scale under synchronous RL, speculative decoding improves rollout throughput by 1.8x. Using a high-fidelity performance simulator, we project that combining speculative decoding with asynchronous RL yields up to 2.5x end-to-end training speedup at 235B scale.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 44 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

38 of 44 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learning26 Jan
  2. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems30 Oct
  3. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning26 Sep
  4. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 29

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  7. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  8. 2025Group Sequence Policy Optimizationcited by 835
  9. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  10. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  11. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  12. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
17 more
  1. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  2. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  3. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  4. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  6. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  7. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 113
  8. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  9. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 42
  10. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  11. 2025RePO: Replay-Enhanced Policy Optimizationcited by 39
  12. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  13. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 32
  14. 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
  15. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  16. 2025ESPO: Entropy Importance Sampling Policy Optimizationcited by 9
  17. 2026QuRL: Efficient Reinforcement Learning with Quantized Rolloutcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Samplingcited by 9
  2. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 6
  3. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 6
  4. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 4
  5. 2026Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Trainingcited by 3
  6. 2026SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rolloutscited by 2
  7. 2026Accelerating Visual On-Policy Distillation with Batched Speculative Jacobi Rolloutscited by 0
  8. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  9. 2026S3R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Training
  2. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
  3. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
  4. 2026When RL Meets Adaptive Speculative Training: A Unified Training-Serving System
  5. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter
  6. 2026SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache
  7. 2026GrowMTP: Can RL Grow Its Own Draft Head?
  8. 2025RLHFSpec: Breaking the Efficiency Bottleneck in RLHF Training via Adaptive Drafting