mlsanityData to 7 Oct 2026

RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs

arXiv 2510.19225 ↗First posted 22 Oct 2025cs.DC cs.LGcited by 10

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has become essential for unlocking advanced reasoning capabilities in large language models (LLMs). RL workflows involve interleaving rollout and training stages with fundamentally different resource requirements. Rollout typically dominates overall execution time, yet scales efficiently through multiple independent instances. In contrast, training requires tightly-coupled GPUs with full-mesh communication. Existing RL frameworks fall into two categories: co-located and disaggregated architectures. Co-located frameworks fail to address this resource tension by forcing both stages to share the same GPUs. Disaggregated architectures, without modifications of well-established RL algorithms, suffer from resource under-utilization. Meanwhile, preemptible GPU resources, i.e., spot instances on public clouds and spare capacity in production clusters, present significant cost-saving opportunities for accelerating RL workflows, if efficiently harvested for rollout. In this paper, we present RLBoost, a framework for cost-efficient RL training that harvests preemptible GPU resources. Our key insight is that rollout's stateless and embarrassingly parallel nature aligns perfectly with preemptible and often fragmented resources. To efficiently utilize these resources despite frequent and unpredictable availability changes, RLBoost adopts a hybrid architecture with three key techniques: (1) adaptive rollout offload to dynamically adjust workloads on the reserved (on-demand) cluster, (2) pull-based weight transfer that quickly provisions newly available instances, and (3) token-level response collection and migration for efficient preemption handling and continuous load balancing. Extensive experiments show RLBoost increases training throughput by 1.51x-1.97x while improving cost efficiency by 28%-49% compared to using only on-demand GPU resources.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 53 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

34 of 53 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation8 Oct
  2. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?1 Oct
  3. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL26 Aug
  4. 2025Geometric-Mean Policy Optimization28 Jul
  5. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  6. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  8. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  9. 2025Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Toolscited by 106
  10. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  11. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  12. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
14 more
  1. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  2. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  3. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 40
  4. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 37
  5. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  6. 2025OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discoverycited by 20
  7. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 19
  8. 2023DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMscited by 17
  9. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  10. 2024Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instancescited by 16
  11. 2025Robust LLM Training Infrastructure at ByteDancecited by 14
  12. 2024Lazarus: Resilient and Elastic Training of Mixture-of-Experts Modelscited by 9
  13. 2020Hoplite: Efficient and Fault-Tolerant Collective Communication for Task-Based Distributed Systemscited by 4
  14. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlapcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  2. 2026TensorHub: Scalable and Elastic Weight Transfer for LLM RL Trainingcited by 4
  3. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 4
  4. 2026Prefix Sliding for efficient test-time scalingcited by 2
  5. 2026AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMscited by 1
  6. 2026LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Servingcited by 1
  7. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 0
  8. 2026Belayer: Efficient Fault Tolerance for LLM Agentic RL Trainingcited by 0
  9. 2026WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learningcited by 0
  10. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  2. 2026RLHFless: Serverless Computing for Efficient RLHF
  3. 2026Performance Foundations of Parallel & Distributed Reasoning Language Models
  4. 2026PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR
  5. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
  6. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs
  7. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scale
  8. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration