mlsanityData to 7 Oct 2026

Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvement

arXiv 2609.22888 ↗First posted 19 Sep 2026cs.ROcited by 0

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Online post-training of vision-language-action (VLA) models requires efficient use of robot interaction and reliable policy improvement from continually collected experience. We propose asynchronous Replay-Anchored Policy improvement (RAPolicy), a framework that performs rollout and learning concurrently while grounding both critic and actor updates in replayed behavior. The critic learns chunk-level values from recorded actions and constructs Bellman targets without predicting next actions, reducing computation and dependence on action-value estimates outside replay coverage. The one-step flow actor reuses the initial noise stored during rollout and learns through advantage-weighted conditional likelihood, directly supervising the action mapping used for execution. We evaluate RAPolicy across four single-task settings and one joint five-task setting in the real world, with online training budgets of only 1--2 hours. Starting from policies fine-tuned on just 10 demonstrations per task, RAPolicy rapidly adapts to new single tasks and achieves an average 86.3% success rate. In the joint multi-task setting, RAPolicy improves overall success rate from 52% to 88% while preserving performance on already reliable tasks and improving weaker capabilities. Overall, RAPolicy substantially outperforms the baselines in aggregate task success while requiring fewer human interventions, demonstrating stable policy improvement and high online training efficiency. Project page: https://flyfaerss.github.io/RAPolicy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 30 of its 31 references that are in the map. First posted on arXiv 19 Sep 2026.

Builds on

30 of 31 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models25 May
  2. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Models24 Apr
  3. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training13 Feb
  4. 2025π*0.6: a VLA That Learns From Experience18 Nov
  5. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning28 Jan

Other references · 25

  1. 2022Flow Matching for Generative Modelingcited by 5,426
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
13 more
  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2018Distributed Prioritized Experience Replaycited by 260
  3. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  4. 2025Flow Q-Learningcited by 162
  5. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  6. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  7. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  8. 2025Reinforcement Learning with Action Chunkingcited by 97
  9. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80
  10. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  11. 2025What Matters for Batch Online Reinforcement Learning in Robotics?cited by 14
  12. 2024Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learningcited by 10
  13. 2026StructRL: Structured Action-Space Exploration for Flow-Based VLAscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SOP: A Scalable Online Post-Training System for Vision-Language-Action Models
  2. 2025Reinforcing Action Policies by Prophesying
  3. 2026VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Models
  4. 2025Interactive Post-Training for Vision-Language-Action Models
  5. 2026Agentic-VLA: Efficient Online Adaptation for Vision-Language-Action Models
  6. 2026ROAD-VLA: Robust Online Adaptation via Self-Distillation for Vision-Language-Action Models
  7. 2025VLA Model Post-Training via Action-Chunked PPO and Self Behavior Cloning
  8. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation