mlsanityData to 7 Oct 2026

π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models

arXiv 2510.25889 ↗First posted 29 Oct 2025cs.LGcited by 16

On the map: Vision-language-action models → Efficient VLA inference and adaptation

Abstract (arXiv)

Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input. Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying RL to large-scale flow-based VLAs (eg, π0, π0.5) remains challenging due to intractable action log-likelihoods raised from flow matching. We address this challenge with π_textttRL, featuring two technical approaches: (1) Flow-Noise models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation. (2) Flow-SDE integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration. We evaluate π_textttRL across various benchmarks, with experiments demonstrating that RL yields significant performance improvements in both in-distribution and out-of-distribution settings.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Efficient VLA inference and adaptation, one of 13 directions in Vision-language-action models. Builds on 48 of its 52 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

48 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Models16 Jan

Other references · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Flow Matching for Generative Modelingcited by 5,426
  10. 2018Neural Ordinary Differential Equationscited by 3,733
  11. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
35 more
  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  6. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  7. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  8. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  10. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  11. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  12. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  13. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  14. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  15. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  16. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  17. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  18. 2024Diffusion Policy Policy Optimizationcited by 255
  19. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  20. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 212
  21. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 168
  22. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  23. 2025Interactive Post-Training for Vision-Language-Action Modelscited by 129
  24. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  25. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 93
  26. 2025Flow Matching Policy Gradientscited by 86
  27. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  28. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  29. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 59
  30. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matchingcited by 55
  31. 2025RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformationcited by 51
  32. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  33. 2025Meta-World+: An Improved, Standardized, RL Benchmarkcited by 37
  34. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Modelscited by 26
  35. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  2. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 28
  3. 2025STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Modelscited by 24
  4. 2026NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Modelscited by 6
  5. 2026WCM: A World Critic Model for Vision-Language-Action Reinforcement Learningcited by 4
  6. 2026WAM-RL: World-Action Model Reinforcement Learning with Reconstruction Rewards and Online Video SFTcited by 3
  7. 2026RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Modelscited by 2
  8. 2026WAM-OPD: On-Policy Distillation for World Action Modelscited by 2
  9. 2026Advances in GRPO for Generation Models: A Surveycited by 1
  10. 2026DEFLECT: Temporal Counterfactual Preference Learning for Delay-Robust Asynchronous VLAscited by 1
  11. 2026Potential-Guided Flow Matching for Vision-Language-Action Policy Improvementcited by 1
  12. 2026VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Modelscited by 0
4 more
  1. 2026HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policycited by 0
  2. 2026Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimizationcited by 0
  3. 2026WAM-OPD: Sharpening World Action Models via On-Policy Distillationcited by 0
  4. 2026Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models
  2. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning
  3. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs
  4. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models
  5. 2026Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning
  6. 2026Reinforcement Learning for Real-Time Vision-Language-Action Policies
  7. 2025Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models
  8. 2026Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models