mlsanityData to 7 Oct 2026

ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy

arXiv 2502.05450 ↗First posted 8 Feb 2025cs.RO cs.AIcited by 139

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Vision-Language-Action (VLA) models have shown substantial potential in real-world robotic manipulation. However, fine-tuning these models through supervised learning struggles to achieve robust performance due to limited, inconsistent demonstrations, especially in contact-rich environments. In this paper, we propose a reinforced fine-tuning approach for VLA models, named ConRFT, which consists of offline and online fine-tuning with a unified consistency-based training objective, to address these challenges. In the offline stage, our method integrates behavior cloning and Q-learning to effectively extract policy from a small set of demonstrations and stabilize value estimating. In the online stage, the VLA model is further fine-tuned via consistency policy, with human interventions to ensure safe exploration and high sample efficiency. We evaluate our approach on eight diverse real-world manipulation tasks. It achieves an average success rate of 96.3% within 45-90 minutes of online fine-tuning, outperforming prior supervised methods with a 144% improvement in success rate and 1.9x shorter episode length. This work highlights the potential of integrating reinforcement learning to enhance the performance of VLA models for real-world robotic applications. Videos and code are available at our project website https://cccedric.github.io/conrft/.

Key facts. Cited by 139 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 50 of its 55 references that are in the map. First posted on arXiv 8 Feb 2025.

Builds on

50 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Octo: An Open-Source Generalist Robot Policy20 May

Other references · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  10. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
37 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  2. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  3. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  4. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  6. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  7. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  8. 2024Iterative Reasoning Preference Optimizationcited by 244
  9. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  10. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  11. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  12. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  13. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  14. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  15. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  16. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  17. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  18. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  19. 2023Robot Parkour Learningcited by 115
  20. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  21. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  22. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  23. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
  24. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  25. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  26. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  27. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  28. 2024RLDG: Robotic Generalist Policy Distillation via Reinforcement Learningcited by 42
  29. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  30. 2021Offline Meta-Reinforcement Learning for Industrial Insertioncited by 37
  31. 2023Boosting Continuous Control with Consistency Policycited by 36
  32. 2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 24
  33. 2024Continuously Improving Mobile Manipulation with Autonomous Real-World RLcited by 20
  34. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learningcited by 13
  35. 2023REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulationcited by 12
  36. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
  37. 2024SELU: Self-Learning Embodied MLLMs in Unknown Environmentscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 54; 2026: 85.

139 papers.

Sort:

By area:

  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  2. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  5. 2025Interactive Post-Training for Vision-Language-Action Modelscited by 129
  6. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  7. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  8. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  9. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  10. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 72
  11. 2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Modelscited by 68
  12. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
38 more
  1. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
  2. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 65
  3. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  4. 2025TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimizationcited by 47
  5. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  6. 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 46
  7. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  8. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  9. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 37
  10. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  11. 2025CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learningcited by 34
  12. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  13. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 29
  14. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
  15. 2025Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Correctionscited by 25
  16. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Surveycited by 25
  17. 2025STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Modelscited by 24
  18. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  19. 2025RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedbackcited by 19
  20. 2025QDepth-VLA: Quantized Depth Prediction as Auxiliary Supervision for Vision-Language-Action Modelscited by 19
  21. 2026SOP: A Scalable Online Post-Training System for Vision-Language-Action Modelscited by 19
  22. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  23. 2025DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Modelscited by 17
  24. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 16
  25. 2026On-the-Fly VLA Adaptation via Test-Time Reinforcement Learningcited by 14
  26. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 12
  27. 2025EvoVLA: Self-Evolving Vision-Language-Action Modelcited by 11
  28. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 11
  29. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  30. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillationcited by 9
  31. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  32. 2025Reflection-Based Task Adaptation for Self-Improving VLAcited by 8
  33. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  34. 2025TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learningcited by 7
  35. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  36. 2026RL-VLA3: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Trainingcited by 7
  37. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
  38. 2026LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoningcited by 7

50 most cited of 139 All 139 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
  2. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning
  3. 2026AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models
  4. 2026What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models
  5. 2026VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models
  6. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation
  7. 2026Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation
  8. 2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models