mlsanityData to 14 Oct 2026

Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving

arXiv 2311.17918 ↗First posted 29 Nov 2023cs.CVcited by 319

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

In autonomous driving, predicting future events in advance and evaluating the foreseeable risks empowers autonomous vehicles to better plan their actions, enhancing safety and efficiency on the road. To this end, we propose Drive-WM, the first driving world model compatible with existing end-to-end planning models. Through a joint spatial-temporal modeling facilitated by view factorization, our model generates high-fidelity multiview videos in driving scenes. Building on its powerful generation ability, we showcase the potential of applying the world model for safe driving planning for the first time. Particularly, our Drive-WM enables driving into multiple futures based on distinct driving maneuvers, and determines the optimal trajectory according to the image-based rewards. Evaluation on real-world driving datasets verifies that our method could generate high-quality, consistent, and controllable multiview videos, opening up possibilities for real-world simulations and safe planning.

Key facts. Cited by 319 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 70 of its 79 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

70 of 79 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  2. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  3. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers31 Mar
  4. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  7. 2022A ConvNet for the 2020scited by 3,579
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
38 more
  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2023Mastering Diverse Domains through World Modelscited by 1,253
  5. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  6. 2022Planning-oriented Autonomous Drivingcited by 1,087
  7. 2020Mastering Atari with Discrete World Modelscited by 1,007
  8. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 984
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  11. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  12. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  13. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  14. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  15. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  16. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  17. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  18. 2016Generating Videos with Scene Dynamicscited by 376
  19. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  20. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  21. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 315
  22. 2022MapTR: Structured Modeling and Learning for Online Vectorized HD Map Constructioncited by 314
  23. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 309
  24. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  25. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  26. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  27. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  28. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generationcited by 259
  29. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 252
  30. 2018Stochastic Video Generation with a Learned Priorcited by 245
  31. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  32. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
  33. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 212
  34. 2019Scaling Autoregressive Video Modelscited by 190
  35. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  36. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  37. 2022Model-Based Imitation Learning for Urban Drivingcited by 150
  38. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 71; 2025: 132; 2026: 113.

319 papers.

Sort:

By area:

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  2. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
  3. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 260
  4. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 191
  5. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 185
  6. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 166
  7. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  8. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  9. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
  10. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  11. 2025Unified Vision-Language-Action Modelcited by 134
  12. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
38 more
  1. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 117
  2. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  3. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 106
  4. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  5. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 95
  6. 2025HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixelscited by 94
  7. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  8. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 92
  9. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  10. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 89
  11. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  12. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  13. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 84
  14. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 83
  15. 2024World Models for Autonomous Driving: An Initial Surveycited by 80
  16. 2025RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learningcited by 79
  17. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 78
  18. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 76
  19. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenescited by 71
  20. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 69
  21. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  22. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 67
  23. 20253D and 4D World Modeling: A Surveycited by 67
  24. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPTcited by 66
  25. 2024World-consistent Video Diffusion with Explicit 3D Modelingcited by 65
  26. 2025A Survey of World Models for Autonomous Drivingcited by 60
  27. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 56
  28. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  29. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 55
  30. 2024FreeVS: Generative View Synthesis on Free Driving Trajectorycited by 53
  31. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Trainingcited by 51
  32. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  33. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  34. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 47
  35. 2026World Model for Robot Learning: A Comprehensive Surveycited by 46
  36. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 45
  37. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 45
  38. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44

50 most cited of 319 All 319 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SV-WAM: An Efficient Surround-View World-Action Model for End-to-End Autonomous Driving
  2. 2026Risk-Controllable Multi-View Diffusion for Driving Scenario Generation
  3. 2026Drive-HWM: Hierarchical World Models for Dynamic-Latent Guided Autonomous Driving
  4. 2026MM-Future: Multi-Mode Joint World-Action Modeling for Autonomous Driving
  5. 2026ReDrive: Shaping Representations with World Modeling for End-to-End Driving
  6. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey
  7. 2025SafeMVDrive: Multi-view Safety-Critical Driving Video Synthesis in the Real World Domain
  8. 2026X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving