mlsanityData to 7 Oct 2026

Improving Vision-Language-Action Model with Online Reinforcement Learning

arXiv 2501.16664 ↗First posted 28 Jan 2025cs.RO cs.CV cs.LGcited by 107

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Recent studies have successfully integrated large vision-language models (VLMs) into low-level robotic control by supervised fine-tuning (SFT) with expert robotic datasets, resulting in what we term vision-language-action (VLA) models. Although the VLA models are powerful, how to improve these large models during interaction with environments remains an open question. In this paper, we explore how to further improve these VLA models via Reinforcement Learning (RL), a commonly used fine-tuning technique for large models. However, we find that directly applying online RL to large VLA models presents significant challenges, including training instability that severely impacts the performance of large models, and computing burdens that exceed the capabilities of most local machines. To address these challenges, we propose iRe-VLA framework, which iterates between Reinforcement Learning and Supervised Learning to effectively improve VLA models, leveraging the exploratory benefits of RL while maintaining the stability of supervised learning. Experiments in two simulated benchmarks and a real-world manipulation suite validate the effectiveness of our method.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 57 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

49 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers12 Sep
  2. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May
  3. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learning5 Feb
  4. 2023Vision-Language Foundation Models as Effective Robot Imitators2 Nov
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  6. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
31 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  10. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  13. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  14. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  15. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  16. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  17. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  18. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  19. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  20. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  21. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  22. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 79
  23. 2023Data Quality in Imitation Learningcited by 72
  24. 2023Learning to Model the World with Languagecited by 66
  25. 2023Collaborating with language models for embodied reasoningcited by 61
  26. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  27. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
  28. 2021Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learningcited by 45
  29. 2023Prompt a Robot to Walk with Large Language Modelscited by 44
  30. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 42
  31. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 50; 2026: 57.

107 papers.

Sort:

By area:

  1. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  2. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
  3. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  4. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  5. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  6. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  7. 2025Interactive Post-Training for Vision-Language-Action Modelscited by 129
  8. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  9. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  10. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  11. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  12. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
38 more
  1. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 65
  2. 2025TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimizationcited by 47
  3. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  4. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
  5. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 39
  6. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 38
  7. 2025Don't Blind Your VLA: Aligning Visual Representations for OOD Generalizationcited by 34
  8. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 28
  9. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Modelscited by 26
  10. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Modelscited by 25
  11. 2025STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Modelscited by 24
  12. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 23
  13. 2025RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Modelscited by 20
  14. 2025EVOLVE-VLA: Test-Time Training from Environment Feedback for Vision-Language-Action Modelscited by 20
  15. 2025RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedbackcited by 19
  16. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  17. 2025DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Modelscited by 17
  18. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  19. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 16
  20. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  21. 2025Reinforcement Learning for Flow-Matching Policiescited by 14
  22. 2025CombatVLA: An Efficient Vision-Language-Action Model for Combat Tasks in 3D Action Role-Playing Gamescited by 13
  23. 2025EvoVLA: Self-Evolving Vision-Language-Action Modelcited by 11
  24. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 11
  25. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 10
  26. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
  27. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  28. 2025When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Modelscited by 9
  29. 2025Reflection-Based Task Adaptation for Self-Improving VLAcited by 8
  30. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  31. 2026FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimizationcited by 8
  32. 2026LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Modelscited by 8
  33. 2026Self-Evolving Embodied Agents via Skill-Harness Evolutioncited by 8
  34. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  35. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Modelscited by 7
  36. 2026RL-VLA3: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Trainingcited by 7
  37. 2026World2Act: Latent Action Post-Training from World Model Dynamicscited by 7
  38. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 7

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models
  2. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy
  3. 2026Reinforcement Learning for Real-Time Vision-Language-Action Policies
  4. 2025Dual-Actor Fine-Tuning of VLA Models: A Talk-and-Tweak Human-in-the-Loop Approach
  5. 2025Teaching RL Agents to Act Better: VLM as Action Advisor for Online Reinforcement Learning
  6. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
  7. 2026Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
  8. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Models