mlsanityData to 7 Oct 2026

ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems

arXiv 2510.26475 ↗First posted 30 Oct 2025cs.LG cs.DCcited by 14

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Adapting large language models (LLMs) via reinforcement learning (RL) is often bottlenecked by the generation stage, which can consume over 75% of the training time. Speculative decoding (SD) accelerates autoregressive generation in serving systems, but its behavior under RL training remains largely unexplored. We identify three critical gaps that hinder the naive integration of SD into RL systems: diminishing speedups at large batch sizes, drafter staleness under continual actor updates, and drafter-induced policy degradation. To address these gaps, we present ReSpec, a system that adapts SD to RL through three complementary mechanisms: dynamically tuning SD configurations, evolving the drafter via knowledge distillation, and weighting updates by rollout rewards. On Qwen models (3B--14B), ReSpec achieves up to 4.5x speedup while preserving reward convergence and training stability, providing a practical solution for efficient RL-based LLM adaptation.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 34 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

29 of 34 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  3. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  5. 2024Decoding Speculative Decoding2 Feb
  6. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  8. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  9. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  10. 2023Online Speculative Decoding11 Oct
  11. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  12. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  10. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  11. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
  12. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
5 more
  1. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  2. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  3. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 35
  4. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 31
  5. 2025DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Trainingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13
  2. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 11
  3. 2026Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Samplingcited by 9
  4. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 8
  5. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  6. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 5
  7. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  8. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 3
  9. 2026Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-trainingcited by 2
  10. 2026Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Trainingcited by 2
  11. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 0
  12. 2026Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowingcited by 0
2 more
  1. 2026Belayer: Efficient Fault Tolerance for LLM Agentic RL Trainingcited by 0
  2. 2026GrowMTP: Can RL Grow Its Own Draft Head?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Training
  2. 2026Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
  3. 2026SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
  4. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
  5. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
  6. 2025RLHFSpec: Breaking the Efficiency Bottleneck in RLHF Training via Adaptive Drafting
  7. 2026Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning
  8. 2026When RL Meets Adaptive Speculative Training: A Unified Training-Serving System