mlsanityData to 14 Oct 2026

RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation

arXiv 2509.15965 ↗First posted 19 Sep 2025cs.LG cs.AI cs.DCcited by 57

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has demonstrated immense potential in advancing artificial general intelligence, agentic intelligence, and embodied intelligence. However, the inherent heterogeneity and dynamicity of RL workflows often lead to low hardware utilization and slow training on existing systems. In this paper, we present RLinf, a high-performance RL training system based on our key observation that the major roadblock to efficient RL training lies in system flexibility. To maximize flexibility and efficiency, RLinf is built atop a novel RL system design paradigm called macro-to-micro flow transformation (M2Flow), which automatically breaks down high-level, easy-to-compose RL workflows at both the temporal and spatial dimensions, and recomposes them into optimized execution flows. Supported by RLinf worker's adaptive communication capability, we devise context switching and elastic pipelining to realize M2Flow transformation, and a profiling-guided scheduling policy to generate optimal execution plans. Extensive evaluations on both reasoning RL and embodied RL tasks demonstrate that RLinf consistently outperforms state-of-the-art systems, achieving 1.07×-2.43× speedup in end-to-end training throughput.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 54 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

31 of 54 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
19 more
  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  3. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  4. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  5. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  6. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  7. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  8. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  9. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  10. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  11. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  12. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
  13. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 108
  14. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  15. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 77
  16. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  17. 2025FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Modelscited by 27
  18. 2019Priority-based Parameter Propagation for Distributed DNN Trainingcited by 21
  19. 2025STAlloc: Enhancing Memory Efficiency in Large-Scale Model Training with Spatio-Temporal Planningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 54.

57 papers.

Sort:

By area:

  1. 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 50
  2. 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 47
  3. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 44
  4. 2026G0.5: One Autoregressive Stream for Robot Reasoning and Actioncited by 18
  5. 2025AsyncVLA: Asynchronous Flow Matching for Vision-Language-Action Modelscited by 17
  6. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 17
  7. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Modelscited by 16
  8. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  9. 2026ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulationcited by 10
  10. 2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Modelscited by 9
  11. 2026ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulationcited by 9
  12. 2026NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Modelscited by 7
38 more
  1. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAscited by 6
  2. 2026IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulationcited by 5
  3. 2026TempoFit: Plug-and-Play Layer-Wise Temporal KV Memory for Long-Horizon Vision-Language-Action Manipulationcited by 5
  4. 2026Pre-VLA: Preemptive Runtime Verification for Reliable Vision-Language-Action and World-Model Rolloutscited by 5
  5. 2026Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memorycited by 5
  6. 2026Sword: Style-Robust World Models as Simulators via Dynamic Latent Bootstrapping for VLA Policy Post-Trainingcited by 4
  7. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
  8. 2026RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policycited by 4
  9. 2026Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructurecited by 3
  10. 2026ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policiescited by 3
  11. 2026TacCoRL: Integrating Tactile Feedback into VLA via Simulationcited by 3
  12. 2026FBFM: A Training-Free Asynchronous Feedback Mechanism for Flow-Matching in World-Action Models Executioncited by 3
  13. 2026Continue or Replan? Bernoulli-Continuation Policy Learning for Adaptive Horizon Executioncited by 3
  14. 2026FiberTune: Preserving Action-Fiber Visual Residuals in Vision-Language-Action Fine-Tuningcited by 2
  15. 2026ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policiescited by 2
  16. 2026PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Modelscited by 2
  17. 2026DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulationcited by 2
  18. 2026DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Trainingcited by 2
  19. 2025RLLaVA: An RL-central Framework for Language and Vision Assistantscited by 1
  20. 2026AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Modelscited by 1
  21. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 1
  22. 2026SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learningcited by 1
  23. 2026Reinforcing VLAs in Task-Agnostic World Modelscited by 1
  24. 2026What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Modelscited by 1
  25. 2026Elastic Queries Reinforcement Learning: Self-Aware Policy Execution for VLA Modelscited by 1
  26. 2026SAPS: Shared Autonomy for Policy Steering by Blending Teleoperation with a Pretrained VLAcited by 1
  27. 2026X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matchingcited by 1
  28. 2026PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloudcited by 1
  29. 2026MagicSim: A Unified Infrastructure for Executable Embodied Interactioncited by 0
  30. 2026Learning More from Less: Reinforcement Learning from Hindsightcited by 0
  31. 2026JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Modelscited by 0
  32. 2026Explicit Kinematic Guidance from Analytic Concepts for Vision-Language-Action Modelscited by 0
  33. 2026Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulationcited by 0
  34. 2026Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulationcited by 0
  35. 2026HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policycited by 0
  36. 2026EBRL: Asynchronous Embodied RL by Multi-Grained Resource Managementcited by 0
  37. 2026Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policycited by 0
  38. 2026RoboFFT: Finetuning generative robot policy via online reinforcement learning with forward processcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs
  2. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
  3. 2025SeamlessFlow: A Trainer Agent Isolation RL Framework Achieving Bubble-Free Pipelines via Tag Scheduling
  4. 2026GFlowRL: Scaling Distribution-Matching RL to Large Language Models
  5. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation
  6. 2026Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning
  7. 2026A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes
  8. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models