mlsanityData to 14 Oct 2026

PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR

arXiv 2605.20863 ↗First posted 20 May 2026cs.DC cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has recently unlocked strong reasoning capabilities in large language models (LLMs), triggering rapid exploration of new algorithms and data. However, RLVR training is notoriously inefficient: long-tailed rollouts, tool-induced stalls, and asymmetric resource requirements between rollout and training introduce substantial idle time that cannot be eliminated by job-local optimizations such as synchronous pipelining, asynchronous rollout, or colocated execution. We argue that this inefficiency is structural. While idle gaps are unavoidable within individual RLVR jobs, they are largely anti-correlated across jobs and therefore exploitable at the cluster level. Leveraging this observation, we present PlexRL, a cluster-level runtime for multiplexing unified LLM services across RLVR jobs. By centrally managing model placement, state transitions, and function-level scheduling under strict affinity constraints, PlexRL time-slices LLM execution across jobs to fill otherwise idle periods without expensive model migration. Our implementation and evaluations demonstrate that PlexRL significantly improves effective cluster capacity and reduces user GPU hour cost by maximum 37.58% while preserving algorithmic flexibility and introducing minimal per-job overhead.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 32 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

20 of 32 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?1 Oct
  2. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts27 Sep
  3. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  4. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  6. 2025s1: Simple test-time scalingcited by 1,341
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  10. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  11. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  12. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
3 more
  1. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  2. 2019Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applicationscited by 17
  3. 2026Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaborationcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RL in the Wild: Characterizing RLVR Training in LLM Deployment
  2. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  3. 2026A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes
  4. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  5. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  6. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling
  7. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
  8. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training