mlsanityData to 14 Oct 2026

SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts

arXiv 2608.04962 ↗First posted 5 Aug 2026cs.LG cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has become a standard approach for improving the reasoning abilities of large language models, but its training is costly because each step requires generating many long reasoning responses. Recent methods use speculative decoding to accelerate the rollouts, where a draft model proposes tokens that the target model verifies. However, we find that the draft model repeatedly misses the same tokens within a response and across responses in the same GRPO group, and the verification feedback on these misses is not used to update the draft during the rollout. Motivated by this analysis, we introduce SPARK, a lightweight self-adaptive mechanism designed to enhance draft models used in speculative GRPO training. SPARK leverages target verification feedback to refine draft predictions during generation while preserving their existing training procedures. Averaged across 21 model-dataset settings, SPARK improves generation speedup by 32.7% and 11.4%, and end-to-end GRPO training speedup by 32.2% and 8.5%, over EAGLE3-style and DFlash-style baselines, respectively.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 34 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

20 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts17 Jun
  2. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation23 Sep
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  9. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  10. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  11. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
  12. 2024Learning Harmonized Representations for Speculative Samplingcited by 72
4 more
  1. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 37
  2. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 9
  3. 2025RLHFSpec: Breaking the Efficiency Bottleneck in RLHF Training via Adaptive Draftingcited by 5
  4. 2026Test-Time Speculationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  2. 2026ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
  2. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
  3. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems
  4. 2026SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache
  5. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Training
  6. 2026S3R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding
  7. 2026GrowMTP: Can RL Grow Its Own Draft Head?
  8. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning