mlsanityData to 14 Oct 2026

3D and 4D World Modeling: A Survey

arXiv 2509.07996 ↗First posted 4 Sep 2025cs.CV cs.ROcited by 67

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

World modeling has become a cornerstone in AI research, enabling agents to understand, represent, and predict the dynamic environments they inhabit. While prior work largely emphasizes generative methods for 2D image and video data, they overlook the rapidly growing body of work that leverages native 3D and 4D representations such as RGB-D imagery, occupancy grids, and LiDAR point clouds for large-scale scene modeling. At the same time, the absence of a standardized definition and taxonomy for "world models" has led to fragmented and sometimes inconsistent claims in the literature. This survey addresses these gaps by presenting the first comprehensive review explicitly dedicated to 3D and 4D world modeling and generation. We establish precise definitions, introduce a structured taxonomy spanning video-based (VideoGen), occupancy-based (OccGen), and LiDAR-based (LiDARGen) approaches, and systematically summarize datasets and evaluation metrics tailored to 3D/4D settings. We further discuss practical applications, identify open challenges, and highlight promising research directions, aiming to provide a coherent and foundational reference for advancing the field. A systematic summary of existing literature is available at https://github.com/worldbench/awesome-3d-4d-world-models

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 312 of its 422 references that are in the map. First posted on arXiv 4 Sep 2025.

Builds on

312 of 422 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025From 2D to 3D Cognition: A Brief Survey of General World Models25 Jun
  2. 2025Dreamland: Controllable World Creation with Simulator and Generative Models9 Jun
  3. 2025NoiseController: Towards Consistent Multi-view Video Generation via Noise Decomposition and Collaboration25 Apr
  4. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness27 Mar
  5. 2025UniFuture: A 4D Driving World Model for Future Generation and Perception17 Mar
  6. 2025Controllable 3D Outdoor Scene Generation via Scene Graphs10 Mar
  7. 2025WorldModelBench: Judging Video Generation Models As World Models28 Feb
  8. 2025GameFactory: Creating New Games with Generative Interactive Videos14 Jan
  9. 2024UrbanWorld: An Urban World Model for 3D City Generation16 Jul
  10. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond6 May
  11. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generation8 Mar
  12. 2024Genie: Generative Interactive Environments23 Feb
9 more
  1. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environments14 Dec
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  3. 2022Scalable Diffusion Models with Transformers19 Dec
  4. 2022Flow Matching for Generative Modeling6 Oct
  5. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  6. 2022MaskGIT: Masked Generative Image Transformer8 Feb
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  8. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  9. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 291

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  12. 2013Auto-Encoding Variational Bayescited by 4,053
38 more
  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  6. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  7. 2024Depth Anything V2cited by 1,422
  8. 2018Demystifying MMD GANscited by 1,293
  9. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  10. 2022Planning-oriented Autonomous Drivingcited by 1,087
  11. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,081
  12. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  13. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  14. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 984
  15. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  16. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  17. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  18. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  19. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 776
  20. 2020Convolutional Occupancy Networkscited by 761
  21. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  22. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  23. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  24. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  25. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  26. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 526
  27. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  28. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  29. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  30. 2016Semantic Scene Completion from a Single Depth Imagecited by 472
  31. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  32. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  33. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  34. 2022Visual Language Maps for Robot Navigationcited by 365
  35. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 363
  36. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 354
  37. 2020Virtual KITTI 2cited by 353
  38. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347

50 most cited of 291 All 291 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 45.

67 papers.

Sort:

By area:

  1. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  2. 2026WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Modelscited by 76
  3. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 33
  4. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  5. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real Worldcited by 23
  6. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  7. 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 21
  8. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  9. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  10. 2025Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresightcited by 14
  11. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Modelscited by 14
  12. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 14
38 more
  1. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 13
  2. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
  3. 2025See4D: Pose-Free 4D Generation via Auto-Regressive Video Inpaintingcited by 12
  4. 20253EED: Ground Everything Everywhere in 3Dcited by 12
  5. 2026Language-Conditioned World Modeling for Visual Navigationcited by 10
  6. 2026Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanationcited by 10
  7. 2025Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedbackcited by 9
  8. 2025Stability Under Scrutiny: Benchmarking Representation Paradigms for Online HD Mappingcited by 7
  9. 2025U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequencescited by 7
  10. 2025DINO-Tok: Adapting DINO for Visual Tokenizerscited by 6
  11. 2026Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rolloutcited by 6
  12. 2026Is Your Driving World Model an All-Around Player?cited by 6
  13. 2026One Video, One World: Turning Monocular Video into Physical 4D Scenescited by 6
  14. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 6
  15. 2025RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generationcited by 5
  16. 2026Aligning Agentic World Models via Knowledgeable Experience Learningcited by 5
  17. 2026SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agentscited by 5
  18. 2025SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Modelcited by 4
  19. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  20. 2026From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generationcited by 4
  21. 2026Discrete-WAM: Unified Discrete Vision-Action Token Editing for World-Policy Learningcited by 4
  22. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  23. 2026PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Spacecited by 4
  24. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  25. 2025Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Modelscited by 3
  26. 2026AdaSFormer: Adaptive Serialized Transformers for Monocular Semantic Scene Completion from Indoor Environmentscited by 3
  27. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  28. 2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applicationscited by 3
  29. 2025Learning to Generate 4D LiDAR Sequencescited by 2
  30. 2026OneWorld: Taming Scene Generation with 3D Unified Representation Autoencodercited by 2
  31. 2026DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Drivingcited by 2
  32. 2026How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoningcited by 2
  33. 2026AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Modelscited by 2
  34. 2026GTA: Advancing Image-to-3D World Generation via Geometry Then Appearance Video Diffusioncited by 2
  35. 2026OmniLiDAR: A Unified Diffusion Framework for Multi-Domain 3D LiDAR Generationcited by 2
  36. 2026Embody4D: A Generalist Data Engine for Embodied 4D World Modelingcited by 1
  37. 2026GEM: Generating LiDAR World Model via Deformable Mambacited by 1
  38. 2026Not All Points Are Equal: Uncertainty-Aware 4D LiDAR Scene Synthesiscited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds
  2. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling
  3. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Models
  4. 2026Beyond Pixel Histories: World Models with Persistent 3D State
  5. 20264DGS-WAM: Bridging Past and Future with an Object-Centric World Action Model based on 4D Gaussian Splatting
  6. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
  7. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control
  8. 2026A Cookbook of 3D Vision: Data, Learning Paradigms, and Application