mlsanityData to 14 Oct 2026

Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey

arXiv 2411.02914 ↗First posted 5 Nov 2024cs.AI cs.CV cs.ROcited by 14

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

World models and video generation are pivotal technologies in the domain of autonomous driving, each playing a critical role in enhancing the robustness and reliability of autonomous systems. World models, which simulate the dynamics of real-world environments, and video generation models, which produce realistic video sequences, are increasingly being integrated to improve situational awareness and decision-making capabilities in autonomous vehicles. This paper investigates the relationship between these two technologies, focusing on how their structural parallels, particularly in diffusion-based models, contribute to more accurate and coherent simulations of driving scenarios. We examine leading works such as JEPA, Genie, and Sora, which exemplify different approaches to world model design, thereby highlighting the lack of a universally accepted definition of world models. These diverse interpretations underscore the field's evolving understanding of how world models can be optimized for various autonomous driving tasks. Furthermore, this paper discusses the key evaluation metrics employed in this domain, such as Chamfer distance for 3D scene reconstruction and Fr'echet Inception Distance (FID) for assessing the quality of generated video content. By analyzing the interplay between video generation and world models, this survey identifies critical challenges and future research directions, emphasizing the potential of these technologies to jointly advance the performance of autonomous driving systems. The findings presented in this paper aim to provide a comprehensive understanding of how the integration of video generation and world models can drive innovation in the development of safer and more reliable autonomous vehicles.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 89 of its 107 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

89 of 107 references.

Closest ancestors · 24

Same direction, newest first

  1. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation17 Oct
  2. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model14 Oct
  3. 2024Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models25 Sep
  4. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving5 Sep
  5. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving26 Aug
  6. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents8 Jul
  7. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecasting12 Jun
  8. 2024Enhancing End-to-End Autonomous Driving with Latent World Model12 Jun
  9. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation3 Jun
  10. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  11. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving7 May
  12. 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
12 more
  1. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
  2. 2024World Models for Autonomous Driving: An Initial Survey5 Mar
  3. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)26 Feb
  4. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Driving29 Dec
  5. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  6. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
  7. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion2 Nov
  8. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  9. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  10. 2023UniWorld: Autonomous Driving Pre-training via World Models14 Aug
  11. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model8 Oct
  12. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 65

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022Scalable Diffusion Models with Transformerscited by 6,054
38 more
  1. 2022Generative Adversarial Networkscited by 5,627
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  7. 2022Video Diffusion Modelscited by 2,305
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  9. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  10. 2018World Modelscited by 1,513
  11. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  12. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  13. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  14. 2023Mastering Diverse Domains through World Modelscited by 1,253
  15. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  16. 2020Mastering Atari with Discrete World Modelscited by 1,007
  17. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  18. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  19. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  20. 2024Genie: Generative Interactive Environmentscited by 747
  21. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  22. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  23. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  24. 2023Instruction Tuning with GPT-4cited by 686
  25. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  26. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  27. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  28. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  29. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  30. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  31. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  32. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  33. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
  34. 2022MAGVIT: Masked Generative Video Transformercited by 346
  35. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  36. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  37. 2023DiffIR: Efficient Diffusion Model for Image Restorationcited by 255
  38. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving
  2. 2025Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models
  3. 2025SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model
  4. 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving
  5. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
  6. 2026Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving
  7. 2026VehDyn: A Driving World Model Benchmark for Vehicle Dynamics
  8. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving