mlsanityData to 14 Oct 2026

Video as the New Language for Real-World Decision Making

arXiv 2402.17139 ↗First posted 27 Feb 2024cs.CV cs.AIcited by 87

Not on the map. Why?

Cited by 26 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Both text and video data are abundant on the internet and support large-scale self-supervised learning through next token or frame prediction. However, they have not been equally leveraged: language models have had significant real-world impact, whereas video generation has remained largely limited to media entertainment. Yet video data captures important information about the physical world that is difficult to express in language. To address this gap, we discuss an under-appreciated opportunity to extend video generation to solve tasks in the real world. We observe how, akin to language, video can serve as a unified interface that can absorb internet knowledge and represent diverse tasks. Moreover, we demonstrate how, like language models, video generation can serve as planners, agents, compute engines, and environment simulators through techniques such as in-context learning, planning and reinforcement learning. We identify major impact opportunities in domains such as robotics, self-driving, and science, supported by recent work that demonstrates how such advanced capabilities in video generation are plausibly within reach. Lastly, we identify key challenges in video generation that mitigate progress. Addressing these challenges will enable video generation models to demonstrate unique value alongside language models in a wider array of AI applications.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). Builds on 72 of its 102 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

72 of 102 references.

References · 72

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  12. 2015VQA: Visual Question Answeringcited by 2,506
38 more
  1. 2022Video Diffusion Modelscited by 2,305
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  10. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  12. 2022Diffusion Models in Vision: A Surveycited by 1,015
  13. 2020Mastering Atari with Discrete World Modelscited by 1,007
  14. 2024Genie: Generative Interactive Environmentscited by 747
  15. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  16. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  17. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  18. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  19. 2023Learning Interactive Real-World Simulatorscited by 458
  20. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  21. 2023Augmented Language Models: a Surveycited by 403
  22. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  23. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  24. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  25. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  26. 2022Visual Prompting via Image Inpaintingcited by 250
  27. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 233
  28. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  29. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  30. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  31. 2023A Survey on Video Diffusion Modelscited by 206
  32. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  33. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  34. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  35. 2023Video Language Planningcited by 162
  36. 2023Vision-Language Models as Success Detectorscited by 142
  37. 2017Procedural Content Generation via Machine Learning (PCGML)cited by 137
  38. 2018Imitating Latent Policies from Observationcited by 101

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 31; 2025: 44; 2026: 12.

87 papers.

Sort:

By area:

  1. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  3. 2024Latent Action Pretraining from Videoscited by 321
  4. 2025Video models are zero-shot learners and reasonerscited by 220
  5. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  6. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  7. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
  8. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  9. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  10. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 130
  11. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  12. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 95
38 more
  1. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 84
  2. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 69
  3. 2024Generative Modeling of Molecular Dynamics Trajectoriescited by 51
  4. 2025PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Dropcited by 50
  5. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
  6. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 47
  7. 2025Pre-Trained Video Generative Models as World Simulatorscited by 44
  8. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44
  9. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 43
  10. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 43
  11. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  12. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  13. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  14. 2024Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamicscited by 28
  15. 2024MarS: a Financial Market Simulation Engine Powered by Generative Foundation Modelcited by 28
  16. 2026World-Gymnast: Training Robots with Reinforcement Learning in a World Modelcited by 27
  17. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 26
  18. 2025Latent Diffusion Planning for Imitation Learningcited by 26
  19. 2024VideoAgent: Self-Improving Video Generationcited by 25
  20. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Modelcited by 25
  21. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 25
  22. 2025Generative Physical AI in Vision: A Surveycited by 24
  23. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  24. 2024GenEx: Generating an Explorable Worldcited by 21
  25. 2025EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memorycited by 20
  26. 2024Controlling the World by Sleight of Handcited by 19
  27. 2025StateSpaceDiffuser: Bringing Long Context to Diffusion World Modelscited by 19
  28. 2025T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generationcited by 19
  29. 2025TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Modelscited by 19
  30. 2025Impossible Videoscited by 16
  31. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  32. 2025Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Modelscited by 14
  33. 2024Learning Truncated Causal History Model for Video Restorationcited by 13
  34. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  35. 2025Solving New Tasks by Adapting Internet Video Knowledgecited by 13
  36. 2025Optimus-3: Dual-Router Aligned Mixture-of-Experts Agent with Dual-Granularity Reasoning-Aware Policy Optimizationcited by 13
  37. 2025The Serial Scaling Hypothesiscited by 13
  38. 2024Generative World Explorercited by 12

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Simulating the Visual World with Artificial Intelligence: A Roadmap
  2. 2025Large Video Planner Enables Generalizable Robot Control
  3. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generation
  4. 2026World Model Self-Distillation: Training World Models to Solve General Tasks
  5. 2026Autonomous Video Generation with Counterfactual Controllability for Self-Evolving World Models
  6. 2026World Action Models: A Survey
  7. 2024RoboDreamer: Learning Compositional World Models for Robot Imagination
  8. 2026Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoning