mlsanityData to 7 Oct 2026

DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation

arXiv 2410.13571 ↗First posted 17 Oct 2024cs.CVcited by 123

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

Closed-loop simulation is essential for advancing end-to-end autonomous driving systems. Contemporary sensor simulation methods, such as NeRF and 3DGS, rely predominantly on conditions closely aligned with training data distributions, which are largely confined to forward-driving scenarios. Consequently, these methods face limitations when rendering complex maneuvers (e.g., lane change, acceleration, deceleration). Recent advancements in autonomous-driving world models have demonstrated the potential to generate diverse driving videos. However, these approaches remain constrained to 2D video generation, inherently lacking the spatiotemporal coherence required to capture intricacies of dynamic driving environments. In this paper, we introduce DriveDreamer4D, which enhances 4D driving scene representation leveraging world model priors. Specifically, we utilize the world model as a data machine to synthesize novel trajectory videos, where structured conditions are explicitly leveraged to control the spatial-temporal consistency of traffic elements. Besides, the cousin data training strategy is proposed to facilitate merging real and synthetic data for optimizing 4DGS. To our knowledge, DriveDreamer4D is the first to utilize video generation models for improving 4D reconstruction in driving scenarios. Experimental results reveal that DriveDreamer4D significantly enhances generation quality under novel trajectory views, achieving a relative improvement in FID by 32.1%, 46.4%, and 16.3% compared to PVG, S3Gaussian, and Deformable-GS. Moreover, DriveDreamer4D markedly enhances the spatiotemporal coherence of driving agents, which is verified by a comprehensive user study and the relative increases of 22.6%, 43.5%, and 15.6% in the NTA-IoU metric.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 82 of its 94 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

82 of 94 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024OmniRe: Omni Urban Scene Reconstruction29 Aug
  2. 2024Dynamic 3D Gaussian Fields for Urban Areas5 Jun
  3. 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving30 May
  4. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  5. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes23 May
  6. 2024SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior29 Mar
  7. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
  8. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  9. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  10. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep

Other references · 72

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  11. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  12. 2022Video Diffusion Modelscited by 2,280
38 more
  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  2. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  5. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  6. 2022Planning-oriented Autonomous Drivingcited by 1,068
  7. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  8. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 753
  9. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  10. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  11. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  12. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  13. 2023Mip-Splatting: Alias-free 3D Gaussian Splattingcited by 661
  14. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  15. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  16. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  17. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  18. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  19. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  20. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  21. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  22. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  23. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  24. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  25. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  26. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  27. 2021Urban Radiance Fieldscited by 263
  28. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  29. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  30. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  31. 2019Differentiable Surface Splatting for Point-based Geometry Processingcited by 238
  32. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  33. 2021HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fieldscited by 210
  34. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  35. 2021PandaSet: Advanced Sensor Suite Dataset for Autonomous Drivingcited by 186
  36. 2023HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Samplingcited by 173
  37. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  38. 2022NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fieldscited by 151

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 64; 2026: 46.

123 papers.

Sort:

By area:

  1. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 178
  2. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  3. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 145
  4. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  5. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  6. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  7. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  8. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  9. 20253D and 4D World Modeling: A Surveycited by 64
  10. 2025A Survey of World Models for Autonomous Drivingcited by 60
  11. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  12. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
38 more
  1. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  2. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  3. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  4. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  5. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  6. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  7. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 35
  8. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  9. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  10. 2025WonderTurbo: Generating Interactive 3D World in 0.72 Secondscited by 31
  11. 2024FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenescited by 30
  12. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  13. 2025EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modelingcited by 29
  14. 2025ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Scene Reconstructioncited by 27
  15. 2025POMATO: Marrying Pointmap Matching with Temporal Motion for Dynamic 3D Reconstructioncited by 24
  16. 2025A Survey of Interactive Generative Videocited by 23
  17. 2025SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overheadcited by 23
  18. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 23
  19. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  20. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  21. 2025HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generationcited by 19
  22. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  23. 2025WorldRFT: Latent World Model Planning with Reinforcement Fine-Tuning for Autonomous Drivingcited by 19
  24. 2024Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Modelcited by 18
  25. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  26. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  27. 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planningcited by 17
  28. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignmentcited by 15
  29. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  30. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  31. 2024EMD: Explicit Motion Modeling for High-Quality Street Gaussian Splattingcited by 13
  32. 2024Extrapolated Urban View Synthesis Benchmarkcited by 13
  33. 2025HumanDreamer-X: Photorealistic Single-image Human Avatars Reconstruction via Gaussian Restorationcited by 13
  34. 2024Driving View Synthesis on Free-form Trajectories with Generative Priorcited by 11
  35. 2025ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidancecited by 11
  36. 2026When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Modelscited by 11
  37. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  38. 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluationcited by 10

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20264D-WAM: 4D Consistent World Modeling for Autonomous Driving
  2. 2024DreamDrive: Generative 4D Scene Modeling from Street View Images
  3. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
  4. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving
  5. 2026VehDyn: A Driving World Model Benchmark for Vehicle Dynamics
  6. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks
  7. 2026X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving
  8. 2026Drive-HWM: Hierarchical World Models for Dynamic-Latent Guided Autonomous Driving