mlsanityData to 14 Oct 2026

What Can RL Bring to VLA Generalization? An Empirical Study

arXiv 2505.19789 ↗First posted 26 May 2025cs.LGcited by 108

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Large Vision-Language Action (VLA) models have shown significant potential for embodied AI. However, their predominant training via supervised fine-tuning (SFT) limits generalization due to susceptibility to compounding errors under distribution shifts. Reinforcement learning (RL) offers a path to overcome these limitations by optimizing for task objectives via trial-and-error, yet a systematic understanding of its specific generalization benefits for VLAs compared to SFT is lacking. To address this, our study introduces a comprehensive benchmark for evaluating VLA generalization and systematically investigates the impact of RL fine-tuning across diverse visual, semantic, and execution dimensions. Our extensive experiments reveal that RL fine-tuning, particularly with PPO, significantly enhances generalization in semantic understanding and execution robustness over SFT, while maintaining comparable visual robustness. We identify PPO as a more effective RL algorithm for VLAs than LLM-derived methods like DPO and GRPO. We also develop a simple recipe for efficient PPO training on VLAs, and demonstrate its practical utility for improving VLA generalization. The project page is at https://rlvla.github.io

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 66 of its 76 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

66 of 76 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy8 Feb
  2. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning28 Jan
  3. 2024GRAPE: Generalizing Robot Policy via Preference Alignment28 Nov
  4. 2024Octo: An Open-Source Generalist Robot Policy20 May

Other references · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  10. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  12. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  13. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  14. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
  15. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  16. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  17. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  18. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  19. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  20. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  21. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  22. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  23. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  24. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 304
  25. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  26. 2019Causal Confusion in Imitation Learningcited by 288
  27. 2024Diffusion Policy Policy Optimizationcited by 280
  28. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  29. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  30. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  31. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  32. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  33. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  34. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  35. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 216
  36. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  37. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  38. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 39; 2026: 69.

108 papers.

Sort:

By area:

  1. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 303
  2. 2025π*0.6: a VLA That Learns From Experiencecited by 303
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  5. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
  6. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  7. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  8. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 67
  9. 2025RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformationcited by 57
  10. 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 50
  11. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 45
  12. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 41
38 more
  1. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 40
  2. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 40
  3. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  4. 2025CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learningcited by 37
  5. 2026World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policycited by 37
  6. 2025Don't Blind Your VLA: Aligning Visual Representations for OOD Generalizationcited by 35
  7. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptancecited by 32
  8. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 31
  9. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Modelscited by 28
  10. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Modelscited by 27
  11. 2025STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Modelscited by 24
  12. 2025RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Modelscited by 23
  13. 2025EVOLVE-VLA: Test-Time Training from Environment Feedback for Vision-Language-Action Modelscited by 22
  14. 2026LIBERO-X: Robustness Litmus for Vision-Language-Action Modelscited by 22
  15. 2026SOP: A Scalable Online Post-Training System for Vision-Language-Action Modelscited by 21
  16. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignmentcited by 21
  17. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 18
  18. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  19. 2025AsyncVLA: Asynchronous Flow Matching for Vision-Language-Action Modelscited by 17
  20. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Modelscited by 16
  21. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
  22. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 15
  23. 2026On-the-Fly VLA Adaptation via Test-Time Reinforcement Learningcited by 14
  24. 2025Affordance Field Intervention: Enabling VLAs to Escape Memory Traps in Robotic Manipulationcited by 13
  25. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 13
  26. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillationcited by 13
  27. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 11
  28. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  29. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  30. 2025Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMscited by 9
  31. 2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Modelscited by 9
  32. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  33. 2025Real-world Reinforcement Learning from Suboptimal Interventionscited by 8
  34. 2026LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoningcited by 8
  35. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  36. 2026RL-VLA3: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Trainingcited by 7
  37. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
  38. 2026NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Modelscited by 7

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization
  2. 2025VLA-R1: Enhancing Reasoning in Vision-Language-Action Models
  3. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  4. 2025Refined Policy Distillation: From VLA Generalists to RL Experts
  5. 2026Enhancing Linguistic Generalization of VLA: Fine-Tuning OpenVLA via Synthetic Instruction Augmentation
  6. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
  7. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  8. 2025RLS3: RL-Based Synthetic Sample Selection to Enhance Spatial Reasoning in Vision-Language Models for Indoor Autonomous Perception