mlsanityData to 14 Oct 2026

Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning

arXiv 2511.14617 ↗First posted 18 Nov 2025cs.DC cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) has emerged as a critical technique for advancing modern Large Language Models (LLMs), yet existing synchronous RL systems face severe performance bottlenecks. The rollout phase, which dominates end-to-end iteration time, suffers from substantial long-tail latency and poor resource utilization due to inherent workload imbalance. We present Seer, a novel context learning RL system that addresses these challenges through a key observation: requests sharing the same prompt exhibit strong similarities in output lengths and response patterns. Leveraging this insight, Seer introduces three coordinated techniques: (1) divided rollout for dynamic load balancing, (2) context-aware scheduling to mitigate long-tail request delays, and (3) adaptive grouped speculative decoding to accelerate generation. These mechanisms work in concert to markedly reduce long-tail latency and improve resource efficiency during rollout. Evaluations on production-grade RL workloads demonstrate that Seer achieves up to 2.04× end-to-end rollout throughput improvement compared to the state-of-the-art synchronous RL systems, while notably reducing long-tail latency by 72-94%.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 45 of its 60 references that are in the map. First posted on arXiv 18 Nov 2025.

Builds on

45 of 60 references.

References · 45

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
33 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  4. 2025Group Sequence Policy Optimizationcited by 835
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  6. 2025Muon is Scalable for LLM Trainingcited by 428
  7. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  8. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  9. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  10. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  11. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  12. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
  13. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
  14. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  15. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 67
  16. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  17. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  18. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  19. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 41
  20. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 39
  21. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 38
  22. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 37
  23. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  24. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 36
  25. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  26. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  27. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 32
  28. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 30
  29. 2025BroRL: Scaling Reinforcement Learning via Broadened Explorationcited by 25
  30. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 22
  31. 2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20
  32. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchronycited by 15
  33. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 24.

25 papers.

Sort:

By area:

  1. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learningcited by 13
  2. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  3. 2026Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Samplingcited by 9
  4. 2026Heddle: A Distributed Orchestration System for Agentic RL Rolloutcited by 8
  5. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Modelscited by 6
  6. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 6
  7. 2026Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learningcited by 5
  8. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  9. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
  10. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Schedulingcited by 4
  11. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 3
  12. 2026Faster Synchronous On-Policy RL via Straggler-Aware Group Sizingcited by 2
13 more
  1. 2026Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-trainingcited by 2
  2. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  3. 2026AREAL-DTA: Dynamic Tree Attention for Efficient Reinforcement Learning of Large Language Modelscited by 1
  4. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 1
  5. 2026Off-Policy Value-Based Reinforcement Learning for Large Language Modelscited by 0
  6. 2026Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Trainingcited by 0
  7. 2026FAST: A Framework for Aligned Sampling and Training in Parallel Reinforcement Learning for Autonomous Drivingcited by 0
  8. 2026TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructurecited by 0
  9. 2026Belayer: Efficient Fault Tolerance for LLM Agentic RL Trainingcited by 0
  10. 2026SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Servingcited by 0
  11. 2026ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learningcited by 0
  12. 2026S3R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decodingcited by 0
  13. 2026Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling
  2. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training
  3. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  4. 2026Scheduling Mixed RL Rollouts Beyond Prefix Locality
  5. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
  6. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
  7. 2025Periodic Asynchrony: An On-Policy Approach for Accelerating LLM Reinforcement Learning
  8. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management