mlsanityData to 7 Oct 2026

RLHFSpec: Breaking the Efficiency Bottleneck in RLHF Training via Adaptive Drafting

arXiv 2512.04752 ↗First posted 4 Dec 2025cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is an important fine-tuning technique for large language models (LLMs) and comprises three stages: generation, inference, and training. The generation stage generates samples that are then used to infer learnable experiences for training. We observe that the generation stage is the bottleneck of the entire execution process and consider it a key point for optimization. Specifically, we realize the first attempt to integrate speculative decoding into the RLHF generation stage and propose RLHFSpec, an RLHF system that accelerates generation execution with efficient speculative decoding and sample reallocation. To fully exploit the performance potential provided by speculative decoding, especially dealing with the dynamic workload of the generation stage, RLHFSpec proposes a workload-aware drafting strategy selection mechanism, which selects the near-optimal strategy by jointly considering the verification cost and the number of accepted tokens. Moreover, RLHFSpec also proposes sample reallocation to fully utilize the GPU resources, and optimizes it with an efficient sample migration mechanism. The experimental results show that the RLHFSpec can achieve higher throughput in the generation stage compared to state-of-the-art works. Moreover, due to the effective alleviation of the generation bottleneck, RLHFSpec also shows significant performance speedup in the entire RLHF execution.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 30 references that are in the map. First posted on arXiv 4 Dec 2025.

Builds on

20 of 30 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Optimizing RLHF Training for Large Language Models with Stage Fusion20 Sep
  2. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training19 Dec
  3. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  6. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  10. 2020Recipes for building an open-domain chatbotcited by 593
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  12. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
4 more
  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  2. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  3. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  4. 2024PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rolloutscited by 7
  2. 2026SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rolloutscited by 1
  3. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 0
  4. 2026Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems
  2. 2026Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
  3. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Training
  4. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter
  5. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation
  6. 2026GrowMTP: Can RL Grow Its Own Draft Head?
  7. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
  8. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts