mlsanityData to 14 Oct 2026

FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning

arXiv 2509.21792 ↗First posted 26 Sep 2025cs.LG cs.AIcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Group relative policy optimization (GRPO) has demonstrated significant potential in improving the reasoning capabilities of large language models (LLMs) via reinforcement learning. However, its practical deployment is impeded by an excessively slow training process, primarily attributed to the computationally intensive autoregressive generation of multiple responses per query, which makes the generation phase the primary performance bottleneck. Although speculative decoding presents a promising direction for acceleration, its direct application in GRPO achieves limited speedup under high-concurrency training conditions. To overcome this limitation, we propose a concurrency-aware speculative decoding framework that dynamically adjusts the drafting and verification strategy according to real-time concurrency levels, thereby maximizing the acceleration of the generation process. Furthermore, to address performance degradation arising from distributional drift between the evolving target model and the fixed draft model during training, we introduce an online draft learning mechanism that enables the draft model to continuously adapt using feedback signals from the target model. Experimental results across multiple mathematical reasoning datasets and models demonstrate that the proposed method achieves end-to-end speedups of 2.35x to 2.72x, significantly surpassing baseline approaches in efficiency. The code is available at https://github.com/yedaotian9/GRPO_speculative.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 51 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

41 of 51 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling24 Aug
  2. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  3. 2025ReCode: Reinforcing Code Generation with Reasoning-Process Rewards7 Aug
  4. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity29 Jul
  5. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback3 Jun
  6. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  7. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  8. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training28 May
  9. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  10. 2025DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning14 May
  11. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  12. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
3 more
  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  11. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  12. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
14 more
  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  3. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  4. 2021Sparse is Enough in Scaling Transformerscited by 86
  5. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 75
  6. 2024Learning Harmonized Representations for Speculative Samplingcited by 72
  7. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  8. 2025LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learningcited by 13
  9. 2025A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiencycited by 12
  10. 2020Hierarchical Roofline Performance Analysis for Deep Learning Applicationscited by 11
  11. 2024Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillationcited by 10
  12. 2025CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Draftercited by 9
  13. 2025Improving LLM-Generated Code Quality with GRPOcited by 8
  14. 2024Clover-2: Accurate Inference for Regressive Lightweight Speculative Decodingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Trainingcited by 10
  2. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 9
  3. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 6
  4. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 4
  5. 2026Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Trainingcited by 3
  6. 2026MotionGRPO: Overcoming Low Intra-Group Diversity in GRPO-Based Egocentric Motion Recoverycited by 1
  7. 2026Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPOcited by 0
  8. 2026S3R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decodingcited by 0
  9. 2026ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
  2. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  3. 2026GRPO Training Dynamics for Small Language Models
  4. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  5. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  6. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding
  7. 2026SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
  8. 2026Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization