mlsanityData to 14 Oct 2026

S3R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding

arXiv 2610.13202 ↗First posted 9 Oct 2026cs.LG cs.CLcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Fast rollout generation for long-context reinforcement learning with verifiable rewards (RLVR) requires reducing KV-cache traffic without changing the response groups used for policy optimization. In our profiled RLVR workloads, attention accounts for 72--76% of autoregressive decoding kernel time. Sparse attention reduces these reads, but sampling directly from a sparse policy changes the response groups used for learning. For group-relative optimization, we show that even exact response-wise reweighting does not generally recover the expected dense-policy update because the advantage of each response depends on the rewards of other group members. We propose Self-Sparse Speculative Rollouts (S3R). It uses sparse attention to propose tokens, then generates responses through full-attention verification and exact speculative sampling. Drafting and verification share the same policy parameter, eliminating the need to train or update a separate drafter. The resulting response groups follow the dense-policy distribution. Under S3R, we characterize speedup conditions and the costs of drafting and verification. We found that dense verification still requires the complete historical KV cache. A KV-aware scheduler therefore limits KV-cache repeated offloading and restoration while maintaining rollout parallelism. % OR thrashing In a matched ablation with an aggressive sparse drafting budget, S3R maintains stable training while direct sparse sampling with training-side correction becomes unstable. On long-context mathematical reasoning, S3R achieves 1.39×~1.35× the end-to-end training throughput of autoregressive rollouts on Qwen3 model, with comparable task performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 53 of its 63 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

53 of 63 references.

Closest ancestors · 24

Same direction, newest first

  1. 2026Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training7 Sep
  2. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding30 Jul
  3. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts17 Jun
  4. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding29 Apr
  5. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention6 Feb
  6. 2026When RL Meets Adaptive Speculative Training: A Unified Training-Serving System6 Feb
  7. 2026DFlash: Block Diffusion for Flash Speculative Decoding5 Feb
  8. 2025Speculative Decoding: Performance or Illusion?31 Dec
  9. 2025RLHFSpec: Breaking the Efficiency Bottleneck in RLHF Training via Adaptive Drafting4 Dec
  10. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding1 Dec
  11. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter20 Nov
  12. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Training17 Nov
12 more
  1. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems30 Oct
  2. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning26 Sep
  3. 2025CLaSp: In-Context Layer Skip for Self-Speculative Decoding30 May
  4. 2025KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimization22 May
  5. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  6. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache5 Feb
  7. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  8. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  9. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  10. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  11. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  12. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  7. 2025Group Sequence Policy Optimizationcited by 835
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  9. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  10. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  11. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  12. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
17 more
  1. 2025Defeating the Training-Inference Mismatch via FP16cited by 66
  2. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
  3. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
  4. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 38
  5. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 30
  6. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  7. 2025Trust Region Masking for Long-Horizon LLM Reinforcement Learningcited by 16
  8. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 14
  9. 2026Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rolloutscited by 9
  10. 2026QuRL: Efficient Reinforcement Learning with Quantized Rolloutcited by 8
  11. 2026QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatchcited by 7
  12. 2026Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspectivecited by 7
  13. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Schedulingcited by 6
  14. 2026SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cachecited by 5
  15. 2026Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learningcited by 5
  16. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 5
  17. 2026AIS: Adaptive Importance Sampling for Quantized RLcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
  2. 2026SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
  3. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
  4. 2026Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
  5. 2026sGPO: Trading Inference FLOPs for Training Efficiency in RLVR
  6. 2026Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
  7. 2026KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization
  8. 2026Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing