Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey
On the map: World models → World models for autonomous driving
also in 1 more
Abstract (arXiv)
World models and video generation are pivotal technologies in the domain of autonomous driving, each playing a critical role in enhancing the robustness and reliability of autonomous systems. World models, which simulate the dynamics of real-world environments, and video generation models, which produce realistic video sequences, are increasingly being integrated to improve situational awareness and decision-making capabilities in autonomous vehicles. This paper investigates the relationship between these two technologies, focusing on how their structural parallels, particularly in diffusion-based models, contribute to more accurate and coherent simulations of driving scenarios. We examine leading works such as JEPA, Genie, and Sora, which exemplify different approaches to world model design, thereby highlighting the lack of a universally accepted definition of world models. These diverse interpretations underscore the field's evolving understanding of how world models can be optimized for various autonomous driving tasks. Furthermore, this paper discusses the key evaluation metrics employed in this domain, such as Chamfer distance for 3D scene reconstruction and Fr'echet Inception Distance (FID) for assessing the quality of generated video content. By analyzing the interplay between video generation and world models, this survey identifies critical challenges and future research directions, emphasizing the potential of these technologies to jointly advance the performance of autonomous driving systems. The findings presented in this paper aim to provide a comprehensive understanding of how the integration of video generation and world models can drive innovation in the development of safer and more reliable autonomous vehicles.
Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 89 of its 107 references that are in the map. First posted on arXiv 5 Nov 2024.
Builds on
89 of 107 references.
Closest ancestors · 24
Same direction, newest first
- 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation17 Oct
- 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model14 Oct
- 2024Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models25 Sep
- 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving5 Sep
- 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving26 Aug
- 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents8 Jul
- 2024UnO: Unsupervised Occupancy Fields for Perception and Forecasting12 Jun
- 2024Enhancing End-to-End Autonomous Driving with Latent World Model12 Jun
- 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation3 Jun
- 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
- 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving7 May
- 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
12 more
- 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
- 2024World Models for Autonomous Driving: An Initial Survey5 Mar
- 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)26 Feb
- 2023Visual Point Cloud Forecasting enables Scalable Autonomous Driving29 Dec
- 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
- 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
- 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion2 Nov
- 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
- 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
- 2023UniWorld: Autonomous Driving Pre-training via World Models14 Aug
- 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model8 Oct
- 2017CARLA: An Open Urban Driving Simulator10 Nov
Other references · 65
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2023GPT-4 Technical Reportcited by 18,396
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2020Denoising Diffusion Implicit Modelscited by 9,781
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
- 2023Visual Instruction Tuningcited by 7,869
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2022Scalable Diffusion Models with Transformerscited by 6,054
38 more
- 2022Generative Adversarial Networkscited by 5,627
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
- 2017Neural Discrete Representation Learningcited by 4,712
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
- 2016Improved Techniques for Training GANscited by 2,970
- 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
- 2022Video Diffusion Modelscited by 2,305
- 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
- 2018World Modelscited by 1,513
- 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
- 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
- 2023Mastering Diverse Domains through World Modelscited by 1,253
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
- 2020Mastering Atari with Discrete World Modelscited by 1,007
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
- 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
- 2024Genie: Generative Interactive Environmentscited by 747
- 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
- 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
- 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
- 2023Instruction Tuning with GPT-4cited by 686
- 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
- 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
- 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
- 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
- 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
- 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
- 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
- 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
- 2022MAGVIT: Masked Generative Video Transformercited by 346
- 2022Flexible Diffusion Modeling of Long Videoscited by 294
- 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
- 2023DiffIR: Efficient Diffusion Model for Image Restorationcited by 255
- 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
50 most cited of 65 All 65 (JSON)
Built on by
Citing papers per year: 2025: 8; 2026: 6.
14 papers.
Sort:
By area:
- 20253D and 4D World Modeling: A Surveycited by 67
- 2025A Survey of World Models for Autonomous Drivingcited by 60
- 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
- 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
- 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
- 2026Flow caching for autoregressive video generationcited by 25
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
- 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
- 2026Motion-Aware Caching for Efficient Autoregressive Video Generationcited by 6
- 2026World Models for Embodied Intelligence: From Plausible to Controllable to Actionablecited by 1
- 2025Weakly-supervised Latent Models for Task-specific Visual-Language Controlcited by 0
- 2026DF3: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigationcited by 0
- 2026JEPA Guided Diffusion: Predictive Vision-Language Conditioning for Generative Traffic Forecasting18 Sep 2026
- 2026World Models for Embodied Intelligence: From Plausible to Controllable to Actionable15 Sep 2026
- 2026From Generation to Simulation: How Far Are World Models from Being True Simulators?24 Aug 2026
- 2026DF3: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation3 Aug 2026
- 2026Motion-Aware Caching for Efficient Autoregressive Video Generation3 May 2026
- 2026Flow caching for autoregressive video generation11 Feb 2026
- 2025Weakly-supervised Latent Models for Task-specific Visual-Language Control23 Nov 2025
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmap11 Nov 2025
- 20253D and 4D World Modeling: A Survey4 Sep 2025
- 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis13 Jun 2025
- 2025Generative AI for Autonomous Driving: Frontiers and Opportunities13 May 2025
- 2025Exploring the Evolution of Physics Cognition in Video Generation: A Survey27 Mar 2025
2 more
- 20253D and 4D World Modeling: A Surveycited by 67
- 2025A Survey of World Models for Autonomous Drivingcited by 60
- 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
- 2026World Models for Embodied Intelligence: From Plausible to Controllable to Actionablecited by 1
- 2025Weakly-supervised Latent Models for Task-specific Visual-Language Controlcited by 0
- 2026DF3: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigationcited by 0
- 2026From Generation to Simulation: How Far Are World Models from Being True Simulators?cited by 0
- 2026World Models for Embodied Intelligence: From Plausible to Controllable to Actionable15 Sep 2026
- 2026From Generation to Simulation: How Far Are World Models from Being True Simulators?24 Aug 2026
- 2026DF3: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation3 Aug 2026
- 2025Weakly-supervised Latent Models for Task-specific Visual-Language Control23 Nov 2025
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmap11 Nov 2025
- 20253D and 4D World Modeling: A Survey4 Sep 2025
- 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Survey14 Feb 2025
- 2025A Survey of World Models for Autonomous Driving20 Jan 2025
- 2026Flow caching for autoregressive video generationcited by 25
- 2026Motion-Aware Caching for Efficient Autoregressive Video Generationcited by 6
- 2026Motion-Aware Caching for Efficient Autoregressive Video Generation3 May 2026
- 2026Flow caching for autoregressive video generation11 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving
- 2025Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models
- 2025SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model
- 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving
- 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
- 2026Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving
- 2026VehDyn: A Driving World Model Benchmark for Vehicle Dynamics
- 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving