mlsanityData to 14 Oct 2026

Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations

arXiv 2412.14803 ↗First posted 19 Dec 2024cs.CV cs.ROcited by 289

On the map: Vision-language-action models → Human-video and latent-action pretraining

Abstract (arXiv)

Visual representations play a crucial role in developing generalist robotic policies. Previous vision encoders, typically pre-trained with single-image reconstruction or two-image contrastive learning, tend to capture static information, often neglecting the dynamic aspects vital for embodied tasks. Recently, video diffusion models (VDMs) demonstrate the ability to predict future frames and showcase a strong understanding of physical world. We hypothesize that VDMs inherently produce visual representations that encompass both current static information and predicted future dynamics, thereby providing valuable guidance for robot action learning. Based on this hypothesis, we propose the Video Prediction Policy (VPP), which learns implicit inverse dynamics model conditioned on predicted future representations inside VDMs. To predict more precise future, we fine-tune pre-trained video foundation model on robot datasets along with internet human manipulation data. In experiments, VPP achieves a 18.6% relative improvement on the Calvin ABC-D generalization benchmark compared to the previous state-of-the-art, and demonstrates a 31.6% increase in success rates for complex real-world dexterous manipulation tasks. Project page at https://video-prediction-policy.github.io

Key facts. Cited by 289 papers in the map (data through 14 Oct 2026). Placed in Human-video and latent-action pretraining, one of 13 directions in Vision-language-action models. Builds on 51 of its 54 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

51 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Latent Action Pretraining from Videos15 Oct

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2022Video Diffusion Modelscited by 2,305
38 more
  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  5. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  13. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  14. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  15. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  16. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  17. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  18. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  19. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  20. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  21. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 349
  22. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  23. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  24. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  25. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  26. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  27. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 214
  28. 2023Language-Driven Representation Learning for Roboticscited by 165
  29. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  30. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  31. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 134
  32. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 130
  33. 2022Offline Visual Representation Learning for Embodied Navigationcited by 75
  34. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 72
  35. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 70
  36. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 70
  37. 2024VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulationcited by 46
  38. 2024Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Controlcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 74; 2026: 215.

289 papers.

Sort:

By area:

  1. 2026World Action Models are Zero-shot Policiescited by 465
  2. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 349
  3. 2026Causal World Modeling for Robot Controlcited by 321
  4. 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 300
  5. 2025Motus: A Unified Latent Action World Modelcited by 251
  6. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  7. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  8. 2025Unified Video Action Modelcited by 205
  9. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  10. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 167
  11. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
  12. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 145
38 more
  1. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  2. 2025GR-3 Technical Reportcited by 111
  3. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 86
  4. 2026WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Modelscited by 76
  5. 2026DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Controlcited by 76
  6. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 75
  7. 2026Unified 4D World Action Modeling from Video Priors with Asynchronous Denoisingcited by 71
  8. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  9. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 68
  10. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 67
  11. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 67
  12. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 64
  13. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 63
  14. 2025Vidar: Embodied Video Diffusion Model for Generalist Manipulationcited by 62
  15. 2025FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policiescited by 57
  16. 2026Motubrain: An Advanced World Action Model for Robot Controlcited by 57
  17. 2025KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skillscited by 52
  18. 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulationcited by 50
  19. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  20. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  21. 2026World Model for Robot Learning: A Comprehensive Surveycited by 46
  22. 2026AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routingcited by 46
  23. 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 45
  24. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44
  25. 2026Hierarchical Planning with Latent World Modelscited by 43
  26. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 39
  27. 2026ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?cited by 38
  28. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 36
  29. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Modelcited by 36
  30. 2026World Guidance: World Modeling in Condition Space for Action Generationcited by 36
  31. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 36
  32. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 36
  33. 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
  34. 2025HiF-VLA: Hindsight, Insight and Foresight through Motion Representation for Vision-Language-Action Modelscited by 32
  35. 2026Self-Supervised Flow Matching for Scalable Multi-Modal Synthesiscited by 31
  36. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 31
  37. 2026BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generationcited by 30
  38. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 29

50 most cited of 289 All 289 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training
  2. 2026Vid2WAM: Distilling Video Diffusion Priors into World Action Models
  3. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation
  4. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations
  5. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs
  6. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learning
  7. 2025ViPRA: Video Prediction for Robot Actions
  8. 2026Being-M0.7: A Latent World-Action Model for Humanoid Robots