mlsanityData to 7 Oct 2026

Sora as a World Model? A Complete Survey on Text-to-Video Generation

arXiv 2403.05131 ↗First posted 8 Mar 2024cs.AI cs.CVcited by 53

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

The evolution of video generation from text, from animating MNIST to simulating the world with Sora, has progressed at a breakneck speed. Here, we systematically discuss how far text-to-video generation technology supports essential requirements in world modeling. We curate 250+ studies on text-based video synthesis and world modeling. We then observe that recent models increasingly support spatial, action, and strategic intelligences in world modeling through adherence to completeness, consistency, invention, as well as human interaction and control. We conclude that text-to-video generation is adept at world modeling, although homework in several aspects, such as the diversity-consistency trade-offs, remains to be addressed.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 115 of its 328 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

115 of 328 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation6 Feb
  2. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens18 Jan
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  4. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  5. 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr
  6. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  7. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 108

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  11. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
38 more
  1. 2016Improved Techniques for Training GANscited by 2,938
  2. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2018World Modelscited by 1,444
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  13. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  14. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  15. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  16. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  17. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  18. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  19. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  20. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  21. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  22. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  23. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  24. 2023Improved Techniques for Training Consistency Modelscited by 430
  25. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  26. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  27. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  28. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  29. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  30. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  31. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  32. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  33. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  34. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  35. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  36. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  37. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  38. 2023Any-to-Any Generation via Composable Diffusioncited by 203

50 most cited of 108 All 108 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 27; 2026: 7.

53 papers.

Sort:

By area:

  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  2. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  3. 20253D and 4D World Modeling: A Surveycited by 64
  4. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 44
  5. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  6. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 30
  7. 2025FilmAgent: A Multi-Agent Framework for End-to-End Film Automation in Virtual 3D Spacescited by 28
  8. 2025CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Modelscited by 26
  9. 2024A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insightscited by 24
  10. 2025Embodied AI: From LLMs to World Modelscited by 23
  11. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generationcited by 21
  12. 2025HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generationcited by 19
38 more
  1. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  2. 2025ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Modelscited by 18
  3. 2025Generative AI for Cel-Animation: A Surveycited by 17
  4. 2024K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferencescited by 16
  5. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  6. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  7. 2024Benchmarking Multi-dimensional AIGC Video Quality Assessment: A Dataset and Unified Modelcited by 13
  8. 2024DreamCinema: Cinematic Transfer with Free Camera and 3D Charactercited by 9
  9. 2024Human-Activity AGV Quality Assessment: A Benchmark Dataset and an Objective Evaluation Metriccited by 9
  10. 2025We'll Fix it in Post: Improving Text-to-Video Generation with Neuro-Symbolic Feedbackcited by 9
  11. 2025A Challenge to Build Neuro-Symbolic Video Agentscited by 5
  12. 2025G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Modelscited by 5
  13. 2026Graph World Models: Concepts, Taxonomy, and Future Directionscited by 5
  14. 2024A Transformer Based Generative Chemical Language AI Model for Structural Elucidation of Organic Compoundscited by 4
  15. 2024Vision Technologies with Applications in Traffic Surveillance Systems: A Holistic Surveycited by 4
  16. 2025Dreamweaver: Learning Compositional World Models from Pixelscited by 4
  17. 2025From 2D to 3D Cognition: A Brief Survey of General World Modelscited by 4
  18. 2025SPARK: Jailbreaking T2V Models by Synergistically Prompting Auditory and Recontextualized Knowledgecited by 4
  19. 2025Counterfactual World Models via Digital Twin-conditioned Video Diffusioncited by 4
  20. 2025PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Modelscited by 4
  21. 2024DeepFake-O-Meter v2.0: An Open Platform for DeepFake Detectioncited by 3
  22. 2024Survey of User Interface Design and Interaction Techniques in Generative AI Applicationscited by 3
  23. 2025A Survey: Spatiotemporal Consistency in Video Generationcited by 3
  24. 2024A Roadmap Towards Automated and Regulated Robotic Systemscited by 2
  25. 2024Sora and V-JEPA Have Not Learned The Complete Real World Model -- A Philosophical Analysis of Video AIs Through the Theory of Productive Imaginationcited by 2
  26. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  27. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  28. 2026Looped World Modelscited by 2
  29. 2025A Self-supervised Motion Representation for Portrait Video Generationcited by 1
  30. 2025PipeDiT: Accelerating Diffusion Transformers in Video Generation with Task Pipelining and Model Decouplingcited by 1
  31. 2026Enhancing Train-Free Infinite-Frame Generation for Consistent Long Videoscited by 1
  32. 2026TunerDiT: Training-free Progressive Steering of Diffusion Transformer for Multi-Event Video Generationcited by 1
  33. 2026Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherencecited by 1
  34. 2024BEND: Bagging Deep Learning Training Based on Efficient Neural Network Diffusioncited by 0
  35. 2024Decoupled Video Generation with Chain of Training-free Diffusion Model Expertscited by 0
  36. 2025BounTCHA: A CAPTCHA Utilizing Boundary Identification in Guided Generative AI-extended Videoscited by 0
  37. 2025MOVi: Training-free Text-conditioned Multi-Object Video Generationcited by 0
  38. 2025Video Understanding by Design: How Datasets Shape Video Modelscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
  2. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms
  3. 2025Yume-1.5: A Text-Controlled Interactive World Generation Model
  4. 2026WorldMark: A Unified Benchmark Suite for Interactive Video World Models
  5. 2024WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text and Image Inputs
  6. 2026World Action Models: A Survey
  7. 2026A Mechanistic View on Video Generation as World Models: State and Dynamics
  8. 2025T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generation