From 2D to 3D Cognition: A Brief Survey of General World Models
On the map: World models → Video-generation world models and simulators
Abstract (arXiv)
World models have garnered increasing attention in the development of artificial general intelligence (AGI), serving as computational frameworks for learning representations of the external world and forecasting future states. While early efforts focused on 2D visual perception and simulation, recent 3D-aware generative world models have demonstrated the ability to synthesize geometrically consistent, interactive 3D environments, marking a shift toward 3D spatial cognition. Despite rapid progress, the field lacks systematic analysis to categorize emerging techniques and clarify their roles in advancing 3D cognitive world models. This survey addresses this need by introducing a conceptual framework, providing a structured and forward-looking review of world models transitioning from 2D perception to 3D cognition. Within this framework, we highlight two key technological drivers, particularly advances in 3D representations and the incorporation of world knowledge, as fundamental pillars. Building on these, we dissect three core cognitive capabilities that underpin 3D world modeling: 3D physical scene generation, 3D spatial reasoning, and 3D spatial interaction. We further examine the deployment of these capabilities in real-world applications, including embodied AI, autonomous driving, digital twin, and gaming/VR. Finally, we identify challenges across data, modeling, and deployment, and outline future directions for advancing more robust and generalizable 3D world models.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 133 of its 182 references that are in the map. First posted on arXiv 25 Jun 2025.
Builds on
133 of 182 references.
Closest ancestors · 6
Same direction, newest first
- 2024UrbanWorld: An Urban World Model for 3D City Generation16 Jul
- 2024Sora as a World Model? A Complete Survey on Text-to-Video Generation8 Mar
- 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
- 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
- 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
- 2017AI2-THOR: An Interactive 3D Environment for Visual AI14 Dec
Other references · 127
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
38 more
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
- 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
- 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
- 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
- 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
- 2020Point Transformercited by 1,462
- 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
- 2018World Modelscited by 1,444
- 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
- 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
- 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2020Mastering Atari with Discrete World Modelscited by 990
- 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
- 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
- 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
- 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
- 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
- 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
- 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
- 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 368
- 2023An Embodied Generalist Agent in 3D Worldcited by 355
- 2022Fast Dynamic Radiance Fields with Time-Aware Neural Voxelscited by 344
- 2023SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenescited by 339
- 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
- 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
- 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
- 2023Photorealistic Video Generation with Diffusion Modelscited by 304
- 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
- 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
- 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 248
- 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
- 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
50 most cited of 127 All 127 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 2.
4 papers.
- 20253D and 4D World Modeling: A Surveycited by 64
- 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
- 2026From Generation to Simulation: How Far Are World Models from Being True Simulators?cited by 0
- 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0
- 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platform1 Oct 2026
- 2026From Generation to Simulation: How Far Are World Models from Being True Simulators?24 Aug 2026
- 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI6 Oct 2025
- 20253D and 4D World Modeling: A Survey4 Sep 2025
Similar, not cited
Close in wording, no citation either way
- 20263D-Belief: Embodied Belief Inference via Generative 3D World Modeling
- 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks
- 2026From Perception to Action: Spatial AI Agents and World Models
- 2025Learning 3D Persistent Embodied World Models
- 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction
- 2025A Comprehensive Survey on World Models for Embodied AI
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
- 20254DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models