mlsanityData to 14 Oct 2026

VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning

arXiv 2505.18719 ↗First posted 24 May 2025cs.RO cs.AIcited by 176

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Recent high-capacity vision-language-action (VLA) models have demonstrated impressive performance on a range of robotic manipulation tasks by imitating human demonstrations. However, exploiting offline data with limited visited states will cause execution failure in out-of-distribution scenarios. Intuitively, an exploration-based method that improves on online collected data at test time could address this limitation. We present VLA-RL, an algorithmic and systematic framework that leverages online reinforcement learning (RL) to improve pretrained auto-regressive VLAs in downstream tasks. Within a unified perspective, we first introduce a trajectory-level RL formulation for auto-regressive VLA training, which models general robotic manipulation trajectory as multi-modal multi-turn conversation. To address the challenge of sparse rewards, we fine-tune a pretrained vision-language model as a robotic process reward model, which is trained on pseudo reward labels annotated on automatically extracted task segments. To scale up, we identify several implementation findings that improve the stability and efficiency including curriculum selection strategy, GPU-balanced vectorized environments, batch decoding, and critic warmup. VLA-RL enables OpenVLA-7B to surpass the strongest finetuned baseline by 4.5% on 40 challenging robotic manipulation tasks in LIBERO, and even matches the performance of advanced commercial models such as π0-FAST. Notably, we observe that VLA-RL benefits from increased test-time optimization, indicating an early spark of inference scaling laws in robotics.

Key facts. Cited by 176 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 80 of its 90 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

80 of 90 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning28 Jan
  2. 2024GRAPE: Generalizing Robot Policy via Preference Alignment28 Nov
  3. 2024Octo: An Open-Source Generalist Robot Policy20 May

Other references · 77

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
38 more
  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  13. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  14. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  15. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  16. 2019Solving Rubik's Cube with a Robot Handcited by 734
  17. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  18. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  19. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  20. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  21. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  22. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  23. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  24. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  25. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  26. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  27. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  28. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  29. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  30. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  31. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  32. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  33. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  34. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  35. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
  36. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  37. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  38. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 59; 2026: 117.

176 papers.

Sort:

By area:

  1. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 303
  2. 2025π*0.6: a VLA That Learns From Experiencecited by 303
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  5. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 179
  6. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
  7. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  8. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 86
  9. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 78
  10. 2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Modelscited by 70
  11. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  12. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 67
38 more
  1. 2025World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Trainingcited by 59
  2. 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 50
  3. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  4. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 47
  5. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 45
  6. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 41
  7. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 40
  8. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 40
  9. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  10. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  11. 2026World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policycited by 37
  12. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  13. 2026CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videoscited by 31
  14. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Modelscited by 28
  15. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Modelscited by 27
  16. 2025STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Modelscited by 24
  17. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 24
  18. 2025RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Modelscited by 23
  19. 2025EVOLVE-VLA: Test-Time Training from Environment Feedback for Vision-Language-Action Modelscited by 22
  20. 2026SOP: A Scalable Online Post-Training System for Vision-Language-Action Modelscited by 21
  21. 2025When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Modelscited by 20
  22. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 19
  23. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 18
  24. 2025DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Modelscited by 18
  25. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  26. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 17
  27. 2026AtomicVLA: Unlocking the Potential of Atomic Skill Learning in Robotscited by 16
  28. 2025Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Modelscited by 15
  29. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 15
  30. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  31. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Modelscited by 14
  32. 2026On-the-Fly VLA Adaptation via Test-Time Reinforcement Learningcited by 14
  33. 2026RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulationcited by 14
  34. 2025VLA-Reasoner: Empowering Vision-Language-Action Models with Reasoning via Online Monte Carlo Tree Searchcited by 13
  35. 2025When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Modelscited by 13
  36. 2025Affordance Field Intervention: Enabling VLAs to Escape Memory Traps in Robotic Manipulationcited by 13
  37. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 13
  38. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillationcited by 13

50 most cited of 176 All 176 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Models
  2. 2026Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
  3. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy
  4. 2026Agentic-VLA: Efficient Online Adaptation for Vision-Language-Action Models
  5. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
  6. 2026AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models
  7. 2026VLaRL: Augmenting Vision-Language-Action Models with Simulation-Trained Latent-Conditioned Residual RL
  8. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulation