mlsanityData to 7 Oct 2026

RL Token: Bootstrapping Online RL with Vision-Language-Action Models

arXiv 2604.23073 ↗First posted 24 Apr 2026cs.LG cs.ROcited by 43

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Vision-language-action (VLA) models can learn to perform diverse manipulation skills "out of the box," but achieving the precision and speed that real-world tasks demand requires further fine-tuning -- for example, via reinforcement learning (RL). We introduce a lightweight method that enables sample-efficient online RL fine-tuning of pretrained VLAs using just a few hours of real-world practice. We (1) adapt the VLA to expose an "RL token," a compact readout representation that preserves task-relevant pretrained knowledge while serving as an efficient interface for online RL, and (2) train a small actor-critic head on this RL token to refine the actions, while anchoring the learned policy to the VLA. Online RL with the RL token (RLT) makes it possible to fine-tune even large VLAs with RL quickly and efficiently. Across four real-robot tasks (screw installation, zip tie fastening, charger insertion, and Ethernet insertion), RLT improves the speed on the hardest part of the task by up to 3x and raises success rates significantly within minutes to a few hours of practice. It can even surpass the speed of human teleoperation on some of the tasks.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 32 of its 45 references that are in the map. First posted on arXiv 24 Apr 2026.

Builds on

32 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025π*0.6: a VLA That Learns From Experience18 Nov
  2. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RL30 Oct
  3. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning11 Sep
  4. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy8 Feb

Other references · 28

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  10. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  11. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  12. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
16 more
  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2018Maximum a Posteriori Policy Optimisationcited by 318
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  4. 2024Diffusion Policy Policy Optimizationcited by 255
  5. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  7. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
  8. 2025Flow Q-Learningcited by 162
  9. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  10. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  11. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  12. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  13. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  14. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  15. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  16. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 43.

43 papers.

Sort:

By area:

  1. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 7
  2. 2026UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptationcited by 5
  3. 2026Trust Region Q Adjoint Matchingcited by 3
  4. 2026TacCoRL: Integrating Tactile Feedback into VLA via Simulationcited by 3
  5. 2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
  6. 2026MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planningcited by 3
  7. 2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 2
  8. 2026ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policiescited by 2
  9. 2026JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignmentcited by 2
  10. 2026DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policiescited by 1
  11. 2026BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Modelscited by 1
  12. 2026dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Modelscited by 1
31 more
  1. 2026Freeform Preference Learning for Robotic Manipulationcited by 1
  2. 2026OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policiescited by 1
  3. 2026HALO-WA: Hybrid-Attention Latent-Guided Online Reinforcement Learning for World-Action Modelscited by 1
  4. 2026Adaptation of Generalist Robot Policies with Minimal Datacited by 1
  5. 2026CounterAlign: Counterfactual Supervision for Vision-Language-Action Modelscited by 1
  6. 2026Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latencycited by 1
  7. 2026PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Modelscited by 1
  8. 2026Reinforcing VLAs in Task-Agnostic World Modelscited by 0
  9. 2026ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learningcited by 0
  10. 2026Prompt-Driven Exploration: Language as an Exploration Space for VLA Reinforcement Learningcited by 0
  11. 2026VINE: Taming Generative Control Policies for Reinforcement Learningcited by 0
  12. 2026Emergent Compositional Skills in Mixture-of-Experts VLAscited by 0
  13. 2026RL2-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Modelscited by 0
  14. 2026X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matchingcited by 0
  15. 2026TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Modelscited by 0
  16. 2026WISE: World-model-guided Imagination Scheduling for Efficient Post-training of Vision-Language-Action Modelscited by 0
  17. 2026VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Modelscited by 0
  18. 2026SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropycited by 0
  19. 2026Reinforcement Learning for Real-Time Vision-Language-Action Policiescited by 0
  20. 2026From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Interventioncited by 0
  21. 2026Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvementcited by 0
  22. 2026Imagine-RL: Residual-Confidence-Guided Cross-Attention for World-Model-Augmented VLA Reinforcement Learningcited by 0
  23. 2026RouteRLT: Learning When and Which RL Specialist Should Control a Vision-Language-Action Policycited by 0
  24. 2026BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Modelscited by 0
  25. 2026VLaRL: Augmenting Vision-Language-Action Models with Simulation-Trained Latent-Conditioned Residual RLcited by 0
  26. 2026Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Modelscited by 0
  27. 2026Principal Steering Subspaces for Online Adaptation of Frozen Generative Robot Policiescited by 0
  28. 2026The Low-Rank Structure of VLA Reinforcement Learningcited by 0
  29. 2026PreferenceFlow: Test-Time Guidance of Flow-Matching Robot Policies from Human Interventionscited by 0
  30. 2026eRLT: Efficient VLA Reinforcement Learning via Action-Relevant Token Routingcited by 0
  31. 2026Q-Learning with Scalar Adjoint Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
  2. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning
  3. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
  4. 2025Teaching RL Agents to Act Better: VLM as Action Advisor for Online Reinforcement Learning
  5. 2026Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
  6. 2026SOP: A Scalable Online Post-Training System for Vision-Language-Action Models
  7. 2026Agentic-VLA: Efficient Online Adaptation for Vision-Language-Action Models
  8. 2026AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models