mlsanityData to 7 Oct 2026

DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation

arXiv 2403.06845 ↗First posted 11 Mar 2024cs.CVcited by 186

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

World models have demonstrated superiority in autonomous driving, particularly in the generation of multi-view driving videos. However, significant challenges still exist in generating customized driving videos. In this paper, we propose DriveDreamer-2, which builds upon the framework of DriveDreamer and incorporates a Large Language Model (LLM) to generate user-defined driving videos. Specifically, an LLM interface is initially incorporated to convert a user's query into agent trajectories. Subsequently, a HDMap, adhering to traffic regulations, is generated based on the trajectories. Ultimately, we propose the Unified Multi-View Model to enhance temporal and spatial coherence in the generated driving videos. DriveDreamer-2 is the first world model to generate customized driving videos, it can generate uncommon driving videos (e.g., vehicles abruptly cut in) in a user-friendly manner. Besides, experimental results demonstrate that the generated videos enhance the training of driving perception methods (e.g., 3D detection and tracking). Furthermore, video generation quality of DriveDreamer-2 surpasses other state-of-the-art methods, showcasing FID and FVD scores of 11.2 and 55.7, representing relative improvements of 30% and 50%.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 64 of its 73 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

64 of 73 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  2. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
  3. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  4. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  5. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model8 Oct

Other references · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
38 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  7. 2020Mastering Atari with Discrete World Modelscited by 990
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  9. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  11. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  12. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  13. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  14. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  15. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  16. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  17. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  18. 2016Generating Videos with Scene Dynamicscited by 376
  19. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  20. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  21. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  22. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  23. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  24. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  25. 2018Stochastic Video Generation with a Learned Priorcited by 244
  26. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  27. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  28. 2019Scaling Autoregressive Video Modelscited by 189
  29. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
  30. 2022Masked World Models for Visual Controlcited by 169
  31. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  32. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  33. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
  34. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 110
  35. 2016Video Pixel Networkscited by 108
  36. 2022TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarioscited by 104
  37. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  38. 2022Deep Hierarchical Planning from Pixelscited by 96

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 86; 2026: 67.

186 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  3. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  4. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
  5. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  6. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  7. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  8. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  9. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  10. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  11. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  12. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
38 more
  1. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  2. 20253D and 4D World Modeling: A Surveycited by 64
  3. 2025A Survey of World Models for Autonomous Drivingcited by 60
  4. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  5. 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Modelcited by 58
  6. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  7. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 54
  8. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  9. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  10. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  11. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  12. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  13. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  14. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  15. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  16. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  17. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  18. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  19. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 30
  20. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  21. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  22. 2024UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Drivingcited by 27
  23. 2025MaskGWM: A Generalizable Driving World Model with Video Mask Reconstructioncited by 27
  24. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Drivingcited by 27
  25. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Modelcited by 25
  26. 2026WorldMark: A Unified Benchmark Suite for Interactive Video World Modelscited by 24
  27. 2025A Survey of Interactive Generative Videocited by 23
  28. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  29. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 23
  30. 2024HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Drivingcited by 21
  31. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  32. 2025UniFuture: A 4D Driving World Model for Future Generation and Perceptioncited by 21
  33. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 21
  34. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  35. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Drivingcited by 20
  36. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latentscited by 19
  37. 2025Enough Coin Flips Can Make LLMs Act Bayesiancited by 19
  38. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19

50 most cited of 186 All 186 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
  2. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
  3. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving
  4. 2024Probing Multimodal LLMs as World Models for Driving
  5. 2025STAGE: A Stream-Centric Generative World Model for Long-Horizon Driving-Scene Simulation
  6. 2024DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving
  7. 2024GenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model
  8. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving