mlsanityData to 7 Oct 2026

A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning

arXiv 2509.15937 ↗First posted 19 Sep 2025cs.RO cs.AIcited by 66

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Robotic real-world reinforcement learning (RL) with vision-language-action (VLA) models is bottlenecked by sparse, handcrafted rewards and inefficient exploration. We introduce VLAC, a general process reward model built upon InternVL and trained on large scale heterogeneous datasets. Given pairwise observations and a language goal, it outputs dense progress delta and done signal, eliminating task-specific reward engineering, and supports one-shot in-context transfer to unseen tasks and environments. VLAC is trained on vision-language datasets to strengthen perception, dialogic and reasoning capabilities, together with robot and human trajectories data that ground action generation and progress estimation, and additionally strengthened to reject irrelevant prompts as well as detect regression or stagnation by constructing large numbers of negative and semantically mismatched samples. With prompt control, a single VLAC model alternately generating reward and action tokens, unifying critic and policy. Deployed inside an asynchronous real-world RL loop, we layer a graded human-in-the-loop protocol (offline demonstration replay, return and explore, human guided explore) that accelerates exploration and stabilizes early learning. Across four distinct real-world manipulation tasks, VLAC lifts success rates from about 30% to about 90% within 200 real-world interaction episodes; incorporating human-in-the-loop interventions yields a further 50% improvement in sample efficiency and achieves up to 100% final success.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 60 of its 63 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

60 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Genie Centurion: Accelerating Scalable Real-World Robot Training with Human Rewind-and-Refine Guidance24 May
  2. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy8 Feb

Other references · 58

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  10. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  11. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  12. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
38 more
  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  2. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  3. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  4. 2025Mean Flows for One-step Generative Modelingcited by 524
  5. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  6. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  7. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  8. 2024Diffusion Policy Policy Optimizationcited by 255
  9. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 191
  10. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  11. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  12. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  13. 2025Flow Q-Learningcited by 162
  14. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  15. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  16. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  17. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
  18. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 120
  19. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 116
  20. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  21. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  22. 2023On Bringing Robots Homecited by 101
  23. 2025Reinforcement Learning with Action Chunkingcited by 97
  24. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  25. 2024Vision Language Models are In-Context Value Learnerscited by 84
  26. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 79
  27. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 70
  28. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  29. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 37
  30. 2025Flow-Based Policy for Online Reinforcement Learningcited by 33
  31. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  32. 2024Adaptive Mobile Manipulation for Articulated Objects In the Open Worldcited by 30
  33. 2024Rank2Reward: Learning Shaped Reward Functions from Passive Videocited by 30
  34. 2023ALAN: Autonomously Exploring Robotic Agents in the Real Worldcited by 23
  35. 2024Practice Makes Perfect: Planning to Learn Skill Parameter Policiescited by 22
  36. 2023Deep RL at Scale: Sorting Waste in Office Buildings with a Fleet of Mobile Manipulatorscited by 21
  37. 2023Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotioncited by 21
  38. 2022Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidancecited by 20

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 58.

66 papers.

Sort:

By area:

  1. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  2. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 65
  3. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  4. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 39
  5. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 37
  6. 2026RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Taskscited by 30
  7. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 28
  8. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
  9. 2025EVOLVE-VLA: Test-Time Training from Environment Feedback for Vision-Language-Action Modelscited by 20
  10. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  11. 2026SOP: A Scalable Online Post-Training System for Vision-Language-Action Modelscited by 19
  12. 2026On-the-Fly VLA Adaptation via Test-Time Reinforcement Learningcited by 14
38 more
  1. 2026PROGRESSLM: Towards Progress Reasoning in Vision-Language Modelscited by 13
  2. 2026chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistenciescited by 11
  3. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 11
  4. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Modelscited by 11
  5. 2026ARM: Advantage Reward Modeling for Long-Horizon Manipulationcited by 10
  6. 2026PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditingcited by 8
  7. 2026ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulationcited by 8
  8. 2025Mind to Hand: Purposeful Robotic Control via Embodied Reasoningcited by 7
  9. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
  10. 2026ABot-Claw: A Foundation for Persistent, Cooperative, and Self-Evolving Robotic Agentscited by 6
  11. 2026PRTS: A Primitive Reasoning and Tasking System via Contrastive Representationscited by 6
  12. 2026ReViP: Mitigating False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalancecited by 5
  13. 2026IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulationcited by 5
  14. 2026From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulationcited by 5
  15. 2026Retrieve-then-Steer: Online Success Memory for Test-Time Adaptation of Generative VLAscited by 5
  16. 2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
  17. 2026ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AIcited by 5
  18. 2026Self-Correcting VLA: Online Action Refinement via Sparse World Imaginationcited by 4
  19. 2026A Progressive Training Strategy for Embodied Vision-Language Models to Mitigate Spatio-Temporal Hallucinationscited by 4
  20. 2026TapSampling: Inference-Time Sampling with a Task-Progress-Understanding Verifier for Robotic Manipulationcited by 4
  21. 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulationcited by 3
  22. 2026Progress Reward Modeling for Robotic Learning: A Comprehensive Surveycited by 3
  23. 2025EVE: A Generator-Verifier System for Generative Policiescited by 2
  24. 2026InSight: Self-Guided Skill Acquisition via Steerable VLAscited by 2
  25. 2026ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuningcited by 2
  26. 2026PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessmentcited by 2
  27. 2026TrAct: Bridging Robot Control and Visual Prediction with Visual Trackscited by 2
  28. 2026Dual-Granularity Contrastive Reward via Generated Episodic Guidance for Efficient Embodied RLcited by 1
  29. 2026SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learningcited by 1
  30. 2026Agentic-VLA: Efficient Online Adaptation for Vision-Language-Action Modelscited by 1
  31. 2026RDA: Reward Design Agent for Reinforcement Learningcited by 1
  32. 2026TACO: TActile World Model as a Self-COrrector for Scalable Robot Policy Post-Trainingcited by 1
  33. 2026ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policiescited by 1
  34. 2026CounterAlign: Counterfactual Supervision for Vision-Language-Action Modelscited by 1
  35. 2026MA-VLCM: A Vision Language Critic Model for Value Estimation of Policies in Multi-Agent Team Settingscited by 0
  36. 2026Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Maskingcited by 0
  37. 2026Feat2Go: Visual Feature-Grounded Value Estimation for Embodied Reinforcement Learningcited by 0
  38. 2026Completion at the Boundary (CaB): Deployable Switching with Completion-Aware Control under Limited Calibrationcited by 0

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
  2. 2025Reinforcing Action Policies by Prophesying
  3. 2026Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models
  4. 2026Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models
  5. 2026Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
  6. 2026Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models
  7. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
  8. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning