mlsanityData to 7 Oct 2026

Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance

arXiv 2410.13816 ↗First posted 17 Oct 2024cs.RO cs.LGcited by 100

On the map: Vision-language-action models → Reinforcement learning and post-training

Abstract (arXiv)

Large, general-purpose robotic policies trained on diverse demonstration datasets have been shown to be remarkably effective both for controlling a variety of robots in a range of different scenes, and for acquiring broad repertoires of manipulation skills. However, the data that such policies are trained on is generally of mixed quality -- not only are human-collected demonstrations unlikely to perform the task perfectly, but the larger the dataset is, the harder it is to curate only the highest quality examples. It also remains unclear how optimal data from one embodiment is for training on another embodiment. In this paper, we present a general and broadly applicable approach that enhances the performance of such generalist robot policies at deployment time by re-ranking their actions according to a value function learned via offline RL. This approach, which we call Value-Guided Policy Steering (V-GPS), is compatible with a wide range of different generalist policies, without needing to fine-tune or even access the weights of the policy. We show that the same value function can improve the performance of five different state-of-the-art policies with different architectures, even though they were trained on distinct datasets, attaining consistent performance improvement on multiple robotic platforms across a total of 12 tasks. Code and videos can be found at: https://nakamotoo.github.io/V-GPS

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 57 of its 62 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

57 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Octo: An Open-Source Generalist Robot Policy20 May

Other references · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  11. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  13. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  14. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  15. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  16. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  17. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  18. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  19. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  20. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  21. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  22. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  23. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  24. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  25. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  26. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  27. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  28. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  29. 2021Offline RL Without Off-Policy Evaluationcited by 150
  30. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  31. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  32. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  33. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  34. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  35. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  36. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  37. 2024The Ingredients for Robotic Diffusion Transformerscited by 72
  38. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 33; 2026: 64.

100 papers.

Sort:

By area:

  1. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  2. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  3. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  4. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 56
  5. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 51
  6. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 48
  7. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
  8. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 39
  9. 2024Inference-Time Policy Steering through Human Interactionscited by 35
  10. 2025CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learningcited by 34
  11. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  12. 2025Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verificationcited by 25
38 more
  1. 2025Inference-Time Enhancement of Generative Robot Policies via Predictive World Modelingcited by 22
  2. 2025Verifier-free Test-Time Sampling for Vision-Language-Action Modelscited by 21
  3. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  4. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  5. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 16
  6. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignmentcited by 15
  7. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14
  8. 2025What Matters for Batch Online Reinforcement Learning in Robotics?cited by 14
  9. 2025Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Modelscited by 14
  10. 2026VLS: Steering Pretrained Robot Policies via Vision-Language Modelscited by 14
  11. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 13
  12. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 13
  13. 2025VLA-Reasoner: Empowering Vision-Language-Action Models with Reasoning via Online Monte Carlo Tree Searchcited by 12
  14. 2026Action Images: End-to-End Policy Learning via Multiview Video Generationcited by 10
  15. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  16. 2026VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Modelcited by 9
  17. 2026Feedback World Model Enables Precise Guidance of Diffusion Policycited by 9
  18. 2025Latent Policy Steering with Embodiment-Agnostic Pretrained World Modelscited by 8
  19. 2025Towards Deploying VLA without Fine-Tuning: Plug-and-Play Inference-Time VLA Policy Steering via Embodied Evolutionary Diffusioncited by 8
  20. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  21. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  22. 2025Learning Affordances at Inference-Time for Vision-Language-Action Modelscited by 6
  23. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 6
  24. 2026OmniGuide: Universal Guidance Fields for Enhancing Generalist Robot Policiescited by 6
  25. 2026Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Trainingcited by 6
  26. 2026VGAS: Value-Guided Action-Chunk Selection for Few-Shot Vision-Language-Action Adaptationcited by 5
  27. 2026You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vectorcited by 5
  28. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 5
  29. 2026APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policiescited by 5
  30. 2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
  31. 2025Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approachcited by 4
  32. 2026Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimizationcited by 4
  33. 2026FASTER: Value-Guided Sampling for Fast RLcited by 4
  34. 2026TapSampling: Inference-Time Sampling with a Task-Progress-Understanding Verifier for Robotic Manipulationcited by 4
  35. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
  36. 2026Guided Action Flow: Q-Guided Inference for Flow-Matching Vision-Language-Action Policiescited by 4
  37. 2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planningcited by 4
  38. 2025Adapting by Analogy: OOD Generalization of Visuomotor Policies via Functional Correspondencecited by 3

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning
  2. 2024STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learning
  3. 2024Effective Tuning Strategies for Generalist Robot Manipulation Policies
  4. 2026World Value Models for Robotic Manipulation
  5. 2026Active Real-World Factor-Based Evaluation for Generalist Robot Policies
  6. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation
  7. 2026ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm
  8. 2026RoboRouter: Training-Free Policy Routing for Robotic Manipulation