mlsanityData to 14 Oct 2026

Video Language Planning

arXiv 2310.10625 ↗First posted 16 Oct 2023cs.CV cs.AI cs.LGcited by 162

Not on the map. Why?

Cited by 59 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data. To this end, we present video language planning (VLP), an algorithm that consists of a tree search procedure, where we train (i) vision-language models to serve as both policies and value functions, and (ii) text-to-video models as dynamics models. VLP takes as input a long-horizon task instruction and current image observation, and outputs a long video plan that provides detailed multimodal (video and language) specifications that describe how to complete the final task. VLP scales with increasing computation budget where more computation time results in improved video plans, and is able to synthesize long-horizon video plans across different robotics domains: from multi-object rearrangement, to multi-camera bi-arm dexterous manipulation. Generated video plans can be translated into real robot actions via goal-conditioned policies, conditioned on each intermediate frame of the generated video. Experiments show that VLP substantially improves long-horizon task success rates compared to prior methods on both simulated and real robots (across 3 hardware platforms).

Key facts. Cited by 162 papers in the map (data through 14 Oct 2026). Builds on 42 of its 52 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

42 of 52 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
30 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  2. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  3. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  5. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  6. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  9. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  10. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  11. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  12. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  13. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 220
  14. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  15. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  16. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  17. 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
  18. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  19. 2023Compositional Foundation Models for Hierarchical Planningcited by 95
  20. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  21. 2021Learning to Compose Visual Relationscited by 72
  22. 2021Controllable and Compositional Generation with Latent-Space Energy-Based Modelscited by 64
  23. 2016Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networkscited by 46
  24. 2021Clockwork Variational Autoencoderscited by 42
  25. 2023Compositional Diffusion-Based Continuous Constraint Solverscited by 40
  26. 2019Model Based Planning with Energy Based Modelscited by 33
  27. 2023Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangementcited by 30
  28. 2023Probabilistic Adaptation of Text-to-Video Modelscited by 29
  29. 2021Composable Energy Policies for Reactive Motion Generation and Reinforcement Learningcited by 16
  30. 2022ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Timecited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 54; 2025: 57; 2026: 48.

162 papers.

Sort:

By area:

  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  3. 2026World Action Models are Zero-shot Policiescited by 465
  4. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  5. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  6. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  7. 2024Latent Action Pretraining from Videoscited by 321
  8. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  9. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  10. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  11. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  12. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
38 more
  1. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  3. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 146
  4. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 142
  5. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
  6. 2025Unified Vision-Language-Action Modelcited by 134
  7. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  8. 2025FLARE: Robot Learning with Implicit World Modelingcited by 113
  9. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  10. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 100
  11. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  12. 2024Video as the New Language for Real-World Decision Makingcited by 87
  13. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  14. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 82
  15. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
  16. 2025Large Video Planner Enables Generalizable Robot Controlcited by 70
  17. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 64
  18. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 64
  19. 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
  20. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 50
  21. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  22. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 49
  23. 2025World-in-World: World Models in a Closed-Loop Worldcited by 47
  24. 2026World Model for Robot Learning: A Comprehensive Surveycited by 46
  25. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformercited by 45
  26. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44
  27. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 43
  28. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 42
  29. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 39
  30. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
  31. 2025Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Processcited by 39
  32. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 38
  33. 2024Learning Manipulation by Predicting Interactioncited by 37
  34. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 37
  35. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 35
  36. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  37. 2025Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planningcited by 34
  38. 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 33

50 most cited of 162 All 162 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models
  2. 2025Image Generation as a Visual Planner for Robotic Manipulation
  3. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Models
  4. 2025Vision-Language-Policy Model for Dynamic Robot Task Planning
  5. 2026Multimodal Behavior Tree Generation: A Small Vision-Language Model for Robot Task Planning
  6. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Tasks
  7. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
  8. 2026CodeGraphVLP: Code-as-Planner Meets Semantic-Graph State for Non-Markovian Vision-Language-Action Models