mlsanityData to 7 Oct 2026

Learning Universal Policies via Text-Guided Video Generation

arXiv 2302.00111 ↗First posted 31 Jan 2023cs.AIcited by 506

Not on the map. Why?

Cited by 151 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A goal of artificial intelligence is to construct an agent that can solve a wide variety of tasks. Recent progress in text-guided image synthesis has yielded models with an impressive ability to generate complex novel images, exhibiting combinatorial generalization across domains. Motivated by this success, we investigate whether such tools can be used to construct more general-purpose agents. Specifically, we cast the sequential decision making problem as a text-conditioned video generation problem, where, given a text-encoded specification of a desired goal, a planner synthesizes a set of future frames depicting its planned actions in the future, after which control actions are extracted from the generated video. By leveraging text as the underlying goal specification, we are able to naturally and combinatorially generalize to novel goals. The proposed policy-as-video formulation can further represent environments with different state and action spaces in a unified space of images, which, for example, enables learning and generalization across a variety of robot manipulation tasks. Finally, by leveraging pretrained language embeddings and widely available videos from the internet, the approach enables knowledge transfer through predicting highly realistic video plans for real robots.

Key facts. Cited by 506 papers in the map (data through 7 Oct 2026). Builds on 36 of its 52 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

36 of 52 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
24 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  7. 2020Mastering Atari with Discrete World Modelscited by 990
  8. 2022A Generalist Agentcited by 802
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  13. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  14. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  15. 2022Transformers are Sample-Efficient World Modelscited by 317
  16. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  17. 2022Multi-Game Decision Transformerscited by 212
  18. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  19. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
  20. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  21. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  22. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  23. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  24. 2019Exponential Family Estimation via Adversarial Dynamics Embeddingcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 41; 2024: 108; 2025: 152; 2026: 205.

506 papers.

Sort:

By area:

  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  3. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  4. 2023Learning Interactive Real-World Simulatorscited by 440
  5. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  6. 2026World Action Models are Zero-shot Policiescited by 411
  7. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  8. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  9. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  10. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  11. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  12. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
38 more
  1. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 311
  2. 2024Latent Action Pretraining from Videoscited by 302
  3. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  4. 2026Causal World Modeling for Robot Controlcited by 289
  5. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  6. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  7. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  8. 2025Motus: A Unified Latent Action World Modelcited by 231
  9. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  10. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  11. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  12. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  13. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  14. 2025Unified Video Action Modelcited by 195
  15. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  16. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  17. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  18. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  19. 2023Video Language Planningcited by 160
  20. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  21. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  22. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  23. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  24. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  25. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  26. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  27. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  28. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  29. 2025Unified Vision-Language-Action Modelcited by 128
  30. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  31. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  32. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  33. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 116
  34. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  35. 2025GR-3 Technical Reportcited by 110
  36. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  37. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  38. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105

50 most cited of 506 All 506 (JSON)