# Vision Language Models are In-Context Value Learners

arXiv: https://arxiv.org/abs/2411.04549 · HTML: https://mlsanity.com/paper/2411.04549/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 61 of its 72 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 61)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [RoboNet: Large-Scale Multi-Robot Learning](https://mlsanity.com/paper/1910.11215/) (2019)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023)
- [LIV: Language-Image Representations and Rewards for Robotic Control](https://mlsanity.com/paper/2306.00958/) (2023)

## Built on by (most cited first, 30 of 90)

- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning](https://mlsanity.com/paper/2509.15937/) (2025)
- [Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons](https://mlsanity.com/paper/2603.02115/) (2026)
- [Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalization](https://mlsanity.com/paper/2505.15660/) (2025)
- [TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics](https://mlsanity.com/paper/2602.19313/) (2026)
- [LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations](https://mlsanity.com/paper/2412.01441/) (2024)
- [Self-Improving Embodied Foundation Models](https://mlsanity.com/paper/2509.15155/) (2025)
- [ViPRA: Video Prediction for Robot Actions](https://mlsanity.com/paper/2511.07732/) (2025)
- [RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Models](https://mlsanity.com/paper/2508.02062/) (2025)
- [EVOLVE-VLA: Test-Time Training from Environment Feedback for Vision-Language-Action Models](https://mlsanity.com/paper/2512.14666/) (2025)
- [BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames](https://mlsanity.com/paper/2602.15010/) (2026)
- [Reinforcement Learning for Flow-Matching Policies](https://mlsanity.com/paper/2507.15073/) (2025)
- [Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations](https://mlsanity.com/paper/2511.22697/) (2025)
- [Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification](https://mlsanity.com/paper/2507.11662/) (2025)
- [PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement](https://mlsanity.com/paper/2411.17764/) (2024)
- [Improving Pre-Trained Vision-Language-Action Policies with Model-Based Search](https://mlsanity.com/paper/2508.12211/) (2025)
- [VLA-Reasoner: Empowering Vision-Language-Action Models with Reasoning via Online Monte Carlo Tree Search](https://mlsanity.com/paper/2509.22643/) (2025)
- [chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistencies](https://mlsanity.com/paper/2602.09021/) (2026)
- [Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models](https://mlsanity.com/paper/2603.16065/) (2026)
- [Addressing the Orchestration Gap in Generalist Robots via Physical Agency](https://mlsanity.com/paper/2607.21725/) (2026)
- [ARM: Advantage Reward Modeling for Long-Horizon Manipulation](https://mlsanity.com/paper/2604.03037/) (2026)
- [Scaffolding Dexterous Manipulation with Vision-Language Models](https://mlsanity.com/paper/2506.19212/) (2025)
- [OpenGVL -- Benchmarking Visual Temporal Progress for Data Curation](https://mlsanity.com/paper/2509.17321/) (2025)
- [Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models](https://mlsanity.com/paper/2512.01946/) (2025)
- [PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing](https://mlsanity.com/paper/2603.21669/) (2026)
- [ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation](https://mlsanity.com/paper/2605.08774/) (2026)
- [Genie Centurion: Accelerating Scalable Real-World Robot Training with Human Rewind-and-Refine Guidance](https://mlsanity.com/paper/2505.18793/) (2025)
- [V0: A Generalist Value Model for Any Policy at State Zero](https://mlsanity.com/paper/2602.03584/) (2026)
- [ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training](https://mlsanity.com/paper/2602.12691/) (2026)

## Abstract (arXiv)

Predicting temporal progress from visual trajectories is important for intelligent robots that can learn, adapt, and improve. However, learning such progress estimator, or temporal value function, across different tasks and domains requires both a large amount of diverse data and methods which can scale and generalize. To address these challenges, we present Generative Value Learning (GVL), a universal value function estimator that leverages the world knowledge embedded in vision-language models (VLMs) to predict task progress. Naively asking a VLM to predict values for a video sequence performs poorly due to the strong temporal correlation between successive frames. Instead, GVL poses value estimation as a temporal ordering problem over shuffled video frames; this seemingly more challenging task encourages VLMs to more fully exploit their underlying semantic and temporal grounding capabilities to differentiate frames based on their perceived task progress, consequently producing significantly better value predictions. Without any robot or task specific training, GVL can in-context zero-shot and few-shot predict effective values for more than 300 distinct real-world tasks across diverse robot platforms, including challenging bimanual manipulation tasks. Furthermore, we demonstrate that GVL permits flexible multi-modal in-context learning via examples from heterogeneous tasks and embodiments, such as human videos. The generality of GVL enables various downstream applications pertinent to visuomotor policy learning, including dataset filtering, success detection, and advantage-weighted regression -- all without any model training or finetuning.