# EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2605.25477 · HTML: https://mlsanity.com/paper/2605.25477/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models.
- Builds on 36 of its 47 references that are in the map.
- First posted on arXiv 25 May 2026.

## Placed in

- Vision-language-action models / [Reinforcement learning and post-training](https://mlsanity.com/vla/reinforcement-learning-and-post-training/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Efficient Online Reinforcement Learning with Offline Data](https://mlsanity.com/paper/2302.02948/) (2023)
- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2410.21845/) (2024)
- [VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning](https://mlsanity.com/paper/2505.18719/) (2025)
- [Steering Your Diffusion Policy with Latent Space Reinforcement Learning](https://mlsanity.com/paper/2506.15799/) (2025)
- [ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy](https://mlsanity.com/paper/2502.05450/) (2025)
- [SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning](https://mlsanity.com/paper/2509.09674/) (2025)
- [Learning a Diffusion Model Policy from Rewards via Q-Score Matching](https://mlsanity.com/paper/2312.11752/) (2023)
- [Interactive Post-Training for Vision-Language-Action Models](https://mlsanity.com/paper/2505.17016/) (2025)
- [Improving Vision-Language-Action Model with Online Reinforcement Learning](https://mlsanity.com/paper/2501.16664/) (2025)
- [Policy Representation via Diffusion Probability Model for Reinforcement Learning](https://mlsanity.com/paper/2305.13122/) (2023)
- [Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control](https://mlsanity.com/paper/2405.16158/) (2024)
- [What Can RL Bring to VLA Generalization? An Empirical Study](https://mlsanity.com/paper/2505.19789/) (2025)
- [Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone](https://mlsanity.com/paper/2412.06685/) (2024)
- [Self-Improving Vision-Language-Action Models with Data Generation via Residual RL](https://mlsanity.com/paper/2511.00091/) (2025)
- [Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model](https://mlsanity.com/paper/2412.13630/) (2024)
- [RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning](https://mlsanity.com/paper/2510.14830/) (2025)
- [Residual Off-Policy RL for Finetuning Behavior Cloning Policies](https://mlsanity.com/paper/2509.19301/) (2025)
- [RL Token: Bootstrapping Online RL with Vision-Language-Action Models](https://mlsanity.com/paper/2604.23073/) (2026)
- [Q-learning with Adjoint Matching](https://mlsanity.com/paper/2601.14234/) (2026)
- [EXPO: Stable Reinforcement Learning with Expressive Policies](https://mlsanity.com/paper/2507.07986/) (2025)

## Built on by (most cited first, 11 of 11)

- [Q-Learning With World Models](https://mlsanity.com/paper/2608.17163/) (2026)
- [Reinforcement Learning for Real-Time Vision-Language-Action Policies](https://mlsanity.com/paper/2609.18207/) (2026)
- [SmoothRL: Online Reinforcement Learning During Asynchronous Execution](https://mlsanity.com/paper/2608.29768/) (2026)
- [VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Models](https://mlsanity.com/paper/2609.04355/) (2026)
- [SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy](https://mlsanity.com/paper/2609.12749/) (2026)
- [From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention](https://mlsanity.com/paper/2609.21788/) (2026)
- [Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvement](https://mlsanity.com/paper/2609.22888/) (2026)
- [Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning](https://mlsanity.com/paper/2610.05882/) (2026)
- [Q-Learning with Scalar Adjoint Matching](https://mlsanity.com/paper/2610.10437/) (2026)
- [Dream to Remember: Continual Steering of Generalist Robot Policies with World Models](https://mlsanity.com/paper/2610.13816/) (2026)
- [ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models](https://mlsanity.com/paper/2610.18596/) (2026)

## Abstract (arXiv)

The ability to efficiently and reliably learn new tasks has been a foundational challenge in robotics. Vision-Language-Action (VLA) models have demonstrated strong generalization across diverse manipulation tasks, yet pretrained policies consistently fall short of the reliability required for real-world deployment. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches either train from scratch without fully leveraging pretrained priors, or fine-tune VLAs without achieving the sample efficiency and success rates that practical deployment demands. We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies that closes this gap. Our system solves a suite of challenging manipulation tasks, including routing string lights and inserting the plug to light it up, striking a pool ball into a pocket, and inserting a flower into a wine bottle, each requiring combinations of high precision, dynamic actions, and robustness to varied initial states. Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data, outperforming both prior RL-from-scratch and VLA finetuning approaches. We release an open-source codebase with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.