mlsanityData to 13 Oct 2026

Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter

arXiv 2511.16665 ↗First posted 20 Nov 2025cs.LG cs.AI cs.DCcited by 13

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

The emergence of Large Language Models (LLMs) with strong reasoning capabilities marks a significant milestone, unlocking new frontiers in complex problem-solving. However, training these reasoning models, typically using Reinforcement Learning (RL), encounters critical efficiency bottlenecks: response generation during RL training exhibits a persistent long-tail distribution, where a few very long responses dominate execution time, wasting resources and inflating costs. To address this, we propose TLT, a system that accelerates reasoning RL training losslessly by integrating adaptive speculative decoding. Applying speculative decoding in RL is challenging due to the dynamic workloads, evolving target model, and draft model training overhead. TLT overcomes these obstacles with two synergistic components: (1) Adaptive Drafter, a lightweight draft model trained continuously on idle GPUs during long-tail generation to maintain alignment with the target model at no extra cost; and (2) Adaptive Rollout Engine, which maintains a memory-efficient pool of pre-captured CUDAGraphs and adaptively select suitable SD strategies for each input batch. Evaluations demonstrate that TLT achieves over 1.7x end-to-end RL training speedup over state-of-the-art systems, preserves the model accuracy, and yields a high-quality draft model as a free byproduct suitable for efficient deployment. Code is released at https://github.com/mit-han-lab/fastrl.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 77 references that are in the map. First posted on arXiv 20 Nov 2025.

Builds on

29 of 77 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training25 Sep
  2. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025Process Reinforcement through Implicit Rewards3 Feb
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  6. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan

Other references · 23

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  2. 2024Qwen2.5 Technical Reportcited by 4,840
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,884
  4. 2025s1: Simple test-time scalingcited by 1,322
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,272
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 990
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 829
  8. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 383
  9. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 319
  10. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  11. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 286
  12. 2023REST: Retrieval-Based Speculative Decodingcited by 148
11 more
  1. 2025Competitive Programming with Large Reasoning Modelscited by 117
  2. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
  3. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  4. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 59
  5. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 59
  6. 2025A Survey on Post-training of Large Language Modelscited by 53
  7. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 35
  8. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 27
  9. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Trainingcited by 13
  10. 2021Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performancecited by 8
  11. 2025AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decodingcited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 10
  2. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  3. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 5
  4. 2026TensorHub: Scalable and Elastic Weight Transfer for LLM RL Trainingcited by 4
  5. 2026Not all tokens are needed(NAT): token efficient reinforcement learningcited by 3
  6. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 3
  7. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learningcited by 2
  8. 2026WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learningcited by 2
  9. 2026AREAL-DTA: Dynamic Tree Attention for Efficient Reinforcement Learning of Large Language Modelscited by 1
  10. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switchcited by 1
  11. 2026Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurationscited by 1
  12. 2026Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Trainingcited by 0
1 more
  1. 2026The Hitchhiker's Guide to Agentic AI: From Foundations to Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Training
  2. 2026Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
  3. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  4. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  5. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
  6. 2025RLHFSpec: Breaking the Efficiency Bottleneck in RLHF Training via Adaptive Drafting
  7. 2025Training Language Models to Reason Efficiently
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Models