mlsanityData to 7 Oct 2026

SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache

arXiv 2601.09083 ↗First posted 14 Jan 2026cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We present Speculative Rollout with Tree-Structured Cache (SRT), a simple, model-free approach to accelerate on-policy reinforcement learning (RL) for language models without sacrificing distributional correctness. SRT exploits the empirical similarity of rollouts for the same prompt across training steps by storing previously generated continuations in a per-prompt tree-structured cache. During generation, the current policy uses this tree as the draft model for performing speculative decoding. To keep the cache fresh and improve draft model quality, SRT updates trees online from ongoing rollouts and proactively performs run-ahead generation during idle GPU bubbles. Integrated into standard RL pipelines (textite.g., PPO, GRPO and DAPO) and multi-turn settings, SRT consistently reduces generation and step latency and lowers per-token inference cost, achieving up to 2.08x wall-clock time speedup during rollout.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 18 of its 21 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

18 of 21 references.

References · 18

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  10. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  11. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  12. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
6 more
  1. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  2. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  3. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  4. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  5. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 35
  6. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 3
  2. 2026Agentic AI Systems Should Be Designed as Marginal Token Allocatorscited by 2
  3. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVRcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
  2. 2026SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
  3. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
  4. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
  5. 2026Speculative Sampling with Reinforcement Learning
  6. 2026GrowMTP: Can RL Grow Its Own Draft Head?
  7. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
  8. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Training