mlsanityData to 14 Oct 2026

Efficient Resource-Shiftable RL Post-training with Generative Rewards

arXiv 2610.16866 ↗First posted 12 Oct 2026cs.LG cs.DCcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF also fits: Process reward models for reasoning

Abstract (arXiv)

Reinforcement learning (RL) post-training with generative reward models (GenRM) has become a central recipe for improving large language models (LLMs), and it is also increasingly resource-intensive: policy rollout, GenRM-based reward evaluation, and policy training all compete for GPU resources. However, existing colocated and disaggregated systems rely on static resource allocation for these roles. Such allocation cannot adapt as each role's workload changes during RL and therefore leaves a role's resources underutilized. We present PivotRL, an RL post-training system that dynamically shifts resources among different roles according to their runtime demands. PivotRL introduces a resource-shiftable runtime built on a Global Parameter Storage, together with a unified mechanism that efficiently shifts role-specific context. A shift-aware scheduler enables sequence migration upon a shift, while a planner considers the aggregate throughput benefit and the shift overhead to determine when and how to shift. We evaluate PivotRL on a 128-GPU cluster using four model groups spanning dense and mixture-of-experts models. PivotRL enables resource shifting within seconds and improves end-to-end throughput by 1.81-2.05 times over colocated deployment and 1.50-1.95 times over disaggregated deployment, while preserving RL convergence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 59 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

40 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging6 Feb
  2. 2026Reward Modeling from Natural Language Human Feedback12 Jan
  3. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models23 Oct

Other references · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  5. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  6. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  7. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  8. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  9. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  10. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  11. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  12. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
25 more
  1. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  2. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  3. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  4. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  5. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  6. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 42
  7. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 41
  8. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  9. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  10. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoningcited by 29
  11. 2025Training AI Co-Scientists Using Rubric Rewardscited by 29
  12. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 16
  13. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchronycited by 15
  14. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 14
  15. 2026StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Trainingcited by 11
  16. 2022Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloadscited by 10
  17. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 9
  18. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Schedulingcited by 6
  19. 2026Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Trainingcited by 6
  20. 2024Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelizationcited by 4
  21. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Schedulingcited by 4
  22. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 3
  23. 2026RubricRM: Generative Reward Modeling via Dynamic Rubrics for Image Generation and Editingcited by 2
  24. 2026Small Language Models as Judges for Rubric-Based Reinforcement Learningcited by 2
  25. 2025Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotationscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Libra: Efficient Resource Management for Agentic RL Post-Training
  2. 2026Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation
  3. 2026GrowMTP: Can RL Grow Its Own Draft Head?
  4. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
  5. 2026PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost
  6. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL
  7. 2026Demystifying Reinforcement Learning Post-Training of Language Models
  8. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation