mlsanityData to 7 Oct 2026

From 2D to 3D Cognition: A Brief Survey of General World Models

arXiv 2506.20134 ↗First posted 25 Jun 2025cs.CVcited by 4

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

World models have garnered increasing attention in the development of artificial general intelligence (AGI), serving as computational frameworks for learning representations of the external world and forecasting future states. While early efforts focused on 2D visual perception and simulation, recent 3D-aware generative world models have demonstrated the ability to synthesize geometrically consistent, interactive 3D environments, marking a shift toward 3D spatial cognition. Despite rapid progress, the field lacks systematic analysis to categorize emerging techniques and clarify their roles in advancing 3D cognitive world models. This survey addresses this need by introducing a conceptual framework, providing a structured and forward-looking review of world models transitioning from 2D perception to 3D cognition. Within this framework, we highlight two key technological drivers, particularly advances in 3D representations and the incorporation of world knowledge, as fundamental pillars. Building on these, we dissect three core cognitive capabilities that underpin 3D world modeling: 3D physical scene generation, 3D spatial reasoning, and 3D spatial interaction. We further examine the deployment of these capabilities in real-world applications, including embodied AI, autonomous driving, digital twin, and gaming/VR. Finally, we identify challenges across data, modeling, and deployment, and outline future directions for advancing more robust and generalizable 3D world models.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 133 of its 182 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

133 of 182 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024UrbanWorld: An Urban World Model for 3D City Generation16 Jul
  2. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generation8 Mar
  3. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  4. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  5. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  6. 2017AI2-THOR: An Interactive 3D Environment for Visual AI14 Dec

Other references · 127

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  9. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  10. 2020Point Transformercited by 1,462
  11. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  12. 2018World Modelscited by 1,444
  13. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  14. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  15. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  16. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  17. 2020Mastering Atari with Discrete World Modelscited by 990
  18. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  19. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  20. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  21. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  22. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  23. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  24. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  25. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  26. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 368
  27. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  28. 2022Fast Dynamic Radiance Fields with Time-Aware Neural Voxelscited by 344
  29. 2023SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenescited by 339
  30. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  31. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  32. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  33. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  34. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  35. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  36. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 248
  37. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  38. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205

50 most cited of 127 All 127 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 20263D-Belief: Embodied Belief Inference via Generative 3D World Modeling
  2. 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks
  3. 2026From Perception to Action: Spatial AI Agents and World Models
  4. 2025Learning 3D Persistent Embodied World Models
  5. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction
  6. 2025A Comprehensive Survey on World Models for Embodied AI
  7. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
  8. 20254DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models