mlsanityData to 7 Oct 2026

Vision Language Models are In-Context Value Learners

arXiv 2411.04549 ↗First posted 7 Nov 2024cs.RO cs.AI cs.LGcited by 84

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Predicting temporal progress from visual trajectories is important for intelligent robots that can learn, adapt, and improve. However, learning such progress estimator, or temporal value function, across different tasks and domains requires both a large amount of diverse data and methods which can scale and generalize. To address these challenges, we present Generative Value Learning (GVL), a universal value function estimator that leverages the world knowledge embedded in vision-language models (VLMs) to predict task progress. Naively asking a VLM to predict values for a video sequence performs poorly due to the strong temporal correlation between successive frames. Instead, GVL poses value estimation as a temporal ordering problem over shuffled video frames; this seemingly more challenging task encourages VLMs to more fully exploit their underlying semantic and temporal grounding capabilities to differentiate frames based on their perceived task progress, consequently producing significantly better value predictions. Without any robot or task specific training, GVL can in-context zero-shot and few-shot predict effective values for more than 300 distinct real-world tasks across diverse robot platforms, including challenging bimanual manipulation tasks. Furthermore, we demonstrate that GVL permits flexible multi-modal in-context learning via examples from heterogeneous tasks and embodiments, such as human videos. The generality of GVL enables various downstream applications pertinent to visuomotor policy learning, including dataset filtering, success detection, and advantage-weighted regression -- all without any model training or finetuning.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 72 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

61 of 72 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Code as Reward: Empowering Reinforcement Learning with VLMs7 Feb
  2. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback6 Feb
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  4. 2023Vision-Language Models as a Source of Rewards14 Dec
  5. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct
  6. 2023Physically Grounded Vision-Language Models for Robotic Manipulation5 Sep
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
38 more
  1. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  6. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  7. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  8. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  9. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  10. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  11. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  12. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  13. 2017One-Shot Imitation Learningcited by 232
  14. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  15. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  16. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  17. 2023Language-Driven Representation Learning for Roboticscited by 163
  18. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  19. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  20. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  21. 2023Vision-Language Models as Success Detectorscited by 138
  22. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  23. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  24. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  25. 2023On Bringing Robots Homecited by 101
  26. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  27. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  28. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  29. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  30. 2020Transformers for One-Shot Visual Imitationcited by 77
  31. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Roboticscited by 71
  32. 2022Zero-Shot Temporal Action Detection via Vision-Language Promptingcited by 65
  33. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  34. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  35. 2024ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperationcited by 61
  36. 2023Language Models as Zero-Shot Trajectory Generatorscited by 59
  37. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
  38. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 41

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 27; 2026: 55.

84 papers.

Sort:

By area:

  1. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  2. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  3. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
  4. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  5. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  6. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  7. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 29
  8. 2025Self-Improving Embodied Foundation Modelscited by 28
  9. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  10. 2025EVOLVE-VLA: Test-Time Training from Environment Feedback for Vision-Language-Action Modelscited by 20
  11. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Modelscited by 19
  12. 2026BPP: Long-Context Robot Imitation Learning by Focusing on Key History Framescited by 17
38 more
  1. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  2. 2025Reinforcement Learning for Flow-Matching Policiescited by 14
  3. 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 12
  4. 2025Improving Pre-Trained Vision-Language-Action Policies with Model-Based Searchcited by 12
  5. 2025VLA-Reasoner: Empowering Vision-Language-Action Models with Reasoning via Online Monte Carlo Tree Searchcited by 12
  6. 2024PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinementcited by 11
  7. 2026chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistenciescited by 11
  8. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Modelscited by 11
  9. 2026ARM: Advantage Reward Modeling for Long-Horizon Manipulationcited by 10
  10. 2025OpenGVL -- Benchmarking Visual Temporal Progress for Data Curationcited by 9
  11. 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Modelscited by 8
  12. 2026PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditingcited by 8
  13. 2026ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulationcited by 8
  14. 2025Genie Centurion: Accelerating Scalable Real-World Robot Training with Human Rewind-and-Refine Guidancecited by 7
  15. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 7
  16. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  17. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
  18. 2026Addressing the Orchestration Gap in Generalist Robots via Physical Agencycited by 7
  19. 2025Learning Affordances at Inference-Time for Vision-Language-Action Modelscited by 6
  20. 2026Recurrent Reasoning with Vision-Language Models for Estimating Long-Horizon Embodied Task Progresscited by 6
  21. 2025PhysiAgent: An Embodied Agent Framework in Physical Worldcited by 5
  22. 2025TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distancecited by 5
  23. 2025Surfer 2: The Next Generation of Cross-Platform Computer Use Agentscited by 5
  24. 2026RynnValue: Scaling Robotic Value Foundation Models with Temporal Distancecited by 5
  25. 2025LuciBot: Automated Robot Policy Learning from Generated Videoscited by 4
  26. 2026Generalizable Dense Reward for Long-Horizon Robotic Taskscited by 3
  27. 2026ProgVLA: Progress-Aware Robot Manipulation Skill Learningcited by 3
  28. 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulationcited by 3
  29. 2026Progress Reward Modeling for Robotic Learning: A Comprehensive Surveycited by 3
  30. 2026Decoding Task Progress from VLA Representationscited by 3
  31. 2025TRACE: A Self-Improving Framework for Robot Behavior Forecasting with Vision-Language Modelscited by 2
  32. 2025EVE: A Generator-Verifier System for Generative Policiescited by 2
  33. 2026SAIL: Test-Time Scaling for In-Context Imitation Learning with VLMcited by 2
  34. 2026Bi-HIL: Bilateral Control-Based Multimodal Hierarchical Imitation Learning via Subtask-Level Progress Rate and Keyframe Memory for Long-Horizon Contact-Rich Robotic Manipulationcited by 2
  35. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulationcited by 2
  36. 2026PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessmentcited by 2
  37. 2025VITA: Zero-Shot Value Functions via Test-Time Adaptation of Vision-Language Modelscited by 1
  38. 2025T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMscited by 1

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ViVa: A Video-Generative Value Model for Robot Reinforcement Learning
  2. 2026World Value Models for Robotic Manipulation
  3. 2026Task Robustness via Re-Labelling Vision-Action Robot Data
  4. 2026Feat2Go: Visual Feature-Grounded Value Estimation for Embodied Reinforcement Learning
  5. 2026Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulation
  6. 2026ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation
  7. 2025Training-free Generation of Temporally Consistent Rewards from VLMs
  8. 2023Video Language Planning