mlsanityData to 7 Oct 2026

DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving

arXiv 2309.09777 ↗First posted 18 Sep 2023cs.CVcited by 258

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

World models, especially in autonomous driving, are trending and drawing extensive attention due to their capacity for comprehending driving environments. The established world model holds immense potential for the generation of high-quality driving videos, and driving policies for safe maneuvering. However, a critical limitation in relevant research lies in its predominant focus on gaming environments or simulated settings, thereby lacking the representation of real-world driving scenarios. Therefore, we introduce DriveDreamer, a pioneering world model entirely derived from real-world driving scenarios. Regarding that modeling the world in intricate driving scenes entails an overwhelming search space, we propose harnessing the powerful diffusion model to construct a comprehensive representation of the complex environment. Furthermore, we introduce a two-stage training pipeline. In the initial phase, DriveDreamer acquires a deep understanding of structured traffic constraints, while the subsequent stage equips it with the ability to anticipate future states. The proposed DriveDreamer is the first world model established from real-world driving scenarios. We instantiate DriveDreamer on the challenging nuScenes benchmark, and extensive experiments verify that DriveDreamer empowers precise, controllable video generation that faithfully captures the structural constraints of real-world traffic scenarios. Additionally, DriveDreamer enables the generation of realistic and reasonable driving policies, opening avenues for interaction and practical applications.

Key facts. Cited by 258 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 70 of its 79 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

70 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model8 Oct

Other references · 69

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  9. 2013Auto-Encoding Variational Bayescited by 3,980
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
38 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  5. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  6. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  7. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  9. 2023Mastering Diverse Domains through World Modelscited by 1,193
  10. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  11. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  12. 2022Planning-oriented Autonomous Drivingcited by 1,068
  13. 2020Mastering Atari with Discrete World Modelscited by 990
  14. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  15. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  16. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  17. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  18. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  19. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  20. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  21. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  22. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  23. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  24. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  25. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  26. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  27. 2016Generating Videos with Scene Dynamicscited by 376
  28. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  29. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  30. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  31. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  32. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  33. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  34. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  35. 2018Stochastic Video Generation with a Learned Priorcited by 244
  36. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  37. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  38. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 86; 2025: 97; 2026: 56.

258 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  3. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  4. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  5. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  6. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  7. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 204
  8. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  9. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 178
  10. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  11. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  12. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
38 more
  1. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  2. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 145
  3. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  4. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  5. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  6. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  7. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  8. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  9. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  10. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  11. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  12. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  13. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  14. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  15. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  16. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  17. 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 91
  18. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  19. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Drivingcited by 89
  20. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  21. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 87
  22. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  23. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  24. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  25. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  26. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  27. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  28. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  29. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenescited by 70
  30. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  31. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  32. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPTcited by 66
  33. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  34. 20253D and 4D World Modeling: A Surveycited by 64
  35. 2025A Survey of World Models for Autonomous Drivingcited by 60
  36. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 59
  37. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  38. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 54

50 most cited of 258 All 258 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving
  2. 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving
  3. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving World
  4. 2026Drive-HWM: Hierarchical World Models for Dynamic-Latent Guided Autonomous Driving
  5. 2025DriveGen: Towards Infinite Diverse Traffic Scenarios with Large Models
  6. 2026X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving
  7. 2026Orbis 2: A Hierarchical World Model for Driving
  8. 2026World Engine: Towards the Era of Post-Training for Autonomous Driving