mlsanityData to 7 Oct 2026

Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond

arXiv 2405.03520 ↗First posted 6 May 2024cs.CVcited by 111

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

General world models represent a crucial pathway toward achieving Artificial General Intelligence (AGI), serving as the cornerstone for various applications ranging from virtual environments to decision-making systems. Recently, the emergence of the Sora model has attained significant attention due to its remarkable simulation capabilities, which exhibits an incipient comprehension of physical laws. In this survey, we embark on a comprehensive exploration of the latest advancements in world models. Our analysis navigates through the forefront of generative methodologies in video generation, where world models stand as pivotal constructs facilitating the synthesis of highly realistic visual content. Additionally, we scrutinize the burgeoning field of autonomous-driving world models, meticulously delineating their indispensable role in reshaping transportation and urban mobility. Furthermore, we delve into the intricacies inherent in world models deployed within autonomous agents, shedding light on their profound significance in enabling intelligent interactions within dynamic environmental contexts. At last, we examine challenges and limitations of world models, and discuss their potential future directions. We hope this survey can serve as a foundational reference for the research community and inspire continued innovation. This survey will be regularly updated at: https://github.com/GigaAI-research/General-World-Models-Survey.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 212 of its 255 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

212 of 255 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
  2. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generation8 Mar
  3. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens18 Jan
  4. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusion12 Dec
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  6. 2023Learning Interactive Real-World Simulators9 Oct
  7. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  8. 2022Scalable Diffusion Models with Transformers19 Dec
  9. 2022MAGVIT: Masked Generative Video Transformer10 Dec
  10. 2022MaskGIT: Masked Generative Image Transformer8 Feb
  11. 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr
  12. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
2 more
  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale22 Oct
  2. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 198

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2017Proximal Policy Optimization Algorithmscited by 14,441
  11. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  12. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
38 more
  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2017Mask R-CNNcited by 7,259
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  13. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  14. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  15. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  16. 2017Neural Discrete Representation Learningcited by 4,650
  17. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  18. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  19. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  20. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  21. 2014Going Deeper with Convolutionscited by 4,231
  22. 2013Auto-Encoding Variational Bayescited by 3,980
  23. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  24. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  25. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  26. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  27. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  28. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  29. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  30. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  31. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  32. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  33. 2022Video Diffusion Modelscited by 2,280
  34. 2018SlowFast Networks for Video Recognitioncited by 1,862
  35. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  36. 2023Consistency Modelscited by 1,778
  37. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  38. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755

50 most cited of 198 All 198 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 45; 2026: 41.

111 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 204
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  4. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  5. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  6. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  7. 2025Video World Models with Long-term Spatial Memorycited by 124
  8. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  9. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 118
  10. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  11. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  12. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformerscited by 77
38 more
  1. 2025Stable Video Infinity: Infinite-Length Video Generation with Error Recyclingcited by 68
  2. 2026WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Modelscited by 68
  3. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  4. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  5. 20253D and 4D World Modeling: A Surveycited by 64
  6. 2025A Survey of World Models for Autonomous Drivingcited by 60
  7. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  8. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 59
  9. 2025Long-Context State-Space Video World Modelscited by 59
  10. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  11. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  12. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  13. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  14. 2024MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generationcited by 41
  15. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
  16. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  17. 2024PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Modelscited by 36
  18. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  19. 2024AVID: Adapting Video Diffusion Models to World Modelscited by 35
  20. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  21. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  22. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  23. 2024MarS: a Financial Market Simulation Engine Powered by Generative Foundation Modelcited by 27
  24. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Modelcited by 25
  25. 2025Generative Physical AI in Vision: A Surveycited by 24
  26. 2025Embodied AI: From LLMs to World Modelscited by 23
  27. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 23
  28. 2025Assessing Adaptive World Models in Machines with Novel Gamescited by 20
  29. 2025HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generationcited by 19
  30. 2024Owl-1: Omni World Model for Consistent Long Video Generationcited by 17
  31. 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Modelscited by 17
  32. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  33. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  34. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  35. 2024ACT-Bench: Towards Action Controllable World Models for Autonomous Drivingcited by 14
  36. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  37. 2024CarDreamer: Open-Source Learning Platform for World Model based Autonomous Drivingcited by 13
  38. 2025HumanDreamer-X: Photorealistic Single-image Human Avatars Reconstruction via Gaussian Restorationcited by 13

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms
  2. 2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications
  3. 2025From 2D to 3D Cognition: A Brief Survey of General World Models
  4. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
  5. 2026A Tutorial on World Models and Physical AI
  6. 2026Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling
  7. 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives
  8. 2026From World Models to World Action Models: A Concise Tutorial for Robotics