mlsanityData to 7 Oct 2026

World Models for Autonomous Driving: An Initial Survey

arXiv 2403.02622 ↗First posted 5 Mar 2024cs.LG cs.AI cs.ROcited by 79

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

In the rapidly evolving landscape of autonomous driving, the capability to accurately predict future events and assess their implications is paramount for both safety and efficiency, critically aiding the decision-making process. World models have emerged as a transformative approach, enabling autonomous driving systems to synthesize and interpret vast amounts of sensor data, thereby predicting potential future scenarios and compensating for information gaps. This paper provides an initial review of the current state and prospective advancements of world models in autonomous driving, spanning their theoretical underpinnings, practical applications, and the ongoing research efforts aimed at overcoming existing limitations. Highlighting the significant role of world models in advancing autonomous driving technologies, this survey aspires to serve as a foundational reference for the research community, facilitating swift access to and comprehension of this burgeoning field, and inspiring continued innovation and exploration.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 63 of its 116 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

63 of 116 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)26 Feb
  2. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  3. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  4. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
  5. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations20 Nov
  6. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  7. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  8. 2023UniWorld: Autonomous Driving Pre-training via World Models14 Aug
  9. 2023TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction7 Mar
  10. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model8 Oct

Other references · 53

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2018World Modelscited by 1,444
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  11. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  12. 2023Mastering Diverse Domains through World Modelscited by 1,193
38 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  2. 2020Mastering Atari with Discrete World Modelscited by 990
  3. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  4. 2018DeepMind Control Suitecited by 935
  5. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  6. 2024Genie: Generative Interactive Environmentscited by 706
  7. 2019Model-Based Reinforcement Learning for Ataricited by 495
  8. 2016Tutorial on Variational Autoencoderscited by 404
  9. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  10. 2022Transformers are Sample-Efficient World Modelscited by 317
  11. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  12. 2021Mastering Atari Games with Limited Datacited by 235
  13. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  14. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  15. 2023Structured World Models from Human Videoscited by 144
  16. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  17. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  18. 2021Discovering and Achieving Goals via World Modelscited by 131
  19. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  20. 2017Visual Interaction Networkscited by 94
  21. 2016The Predictron: End-To-End Learning and Planningcited by 92
  22. 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learningcited by 88
  23. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 87
  24. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
  25. 2017Recurrent Environment Simulatorscited by 68
  26. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  27. 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 53
  28. 2015On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Modelscited by 52
  29. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Modelscited by 51
  30. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
  31. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  32. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 50
  33. 2015The social dilemma of autonomous vehiclescited by 47
  34. 2021Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systemscited by 36
  35. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  36. 2023GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Modelscited by 31
  37. 2022DreamingV2: Reinforcement Learning with Discrete World Models without Reconstructioncited by 29
  38. 2020Active World Model Learning with Progress Curiositycited by 27

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 37; 2026: 27.

79 papers.

Sort:

By area:

  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  2. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  3. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPTcited by 66
  4. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  5. 20253D and 4D World Modeling: A Surveycited by 64
  6. 2025A Survey of World Models for Autonomous Drivingcited by 60
  7. 2024A Survey of Mambacited by 39
  8. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  9. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  10. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  11. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  12. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
38 more
  1. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Modelcited by 25
  2. 2025Text2World: Benchmarking Large Language Models for Symbolic World Model Generationcited by 25
  3. 2025Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Drivingcited by 24
  4. 2025UniFuture: A 4D Driving World Model for Future Generation and Perceptioncited by 21
  5. 2025HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generationcited by 19
  6. 2025WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planningcited by 19
  7. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  8. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignmentcited by 15
  9. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  10. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  11. 2026iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Frameworkcited by 14
  12. 2024CarDreamer: Open-Source Learning Platform for World Model based Autonomous Drivingcited by 13
  13. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
  14. 2024Real-time Accident Anticipation for Autonomous Driving Through Monocular Depth-Enhanced 3D Modelingcited by 12
  15. 2025World Models for Cognitive Agents: Transforming Edge Intelligence in Future Networkscited by 12
  16. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 12
  17. 2024Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Modelscited by 10
  18. 2025Closing the Train-Test Gap in World Models for Gradient-Based Planningcited by 10
  19. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 10
  20. 2025AirScape: An Aerial Generative World Model with Motion Controllabilitycited by 7
  21. 2025World Model-Based End-to-End Scene Generation for Accident Anticipation in Autonomous Drivingcited by 7
  22. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 6
  23. 2026GraphWorld: Long-Horizon Planning with World Models for End-to-End Autonomous Drivingcited by 6
  24. 2025STAGE: A Stream-Centric Generative World Model for Long-Horizon Driving-Scene Simulationcited by 5
  25. 2025The Safety Challenge of World Models for Embodied AI Agents: A Reviewcited by 5
  26. 2026Aligning Agentic World Models via Knowledgeable Experience Learningcited by 5
  27. 2026Graph World Models: Concepts, Taxonomy, and Future Directionscited by 5
  28. 2024When, Where, and What? A Novel Benchmark for Accident Anticipation and Localization with Large Language Modelscited by 4
  29. 2025Beyond Patterns: Harnessing Causal Logic for Autonomous Driving Trajectory Predictioncited by 4
  30. 2025Time-Aware World Model for Adaptive Prediction and Controlcited by 4
  31. 2026MRI Contrast Enhancement Kinetics World Modelcited by 4
  32. 2026Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Drivingcited by 4
  33. 2024"Sora is Incredible and Scary": Emerging Governance Challenges of Text-to-Video Generative AI Modelscited by 3
  34. 2025Edge General Intelligence Through World Models and Agentic AI: Fundamentals, Solutions, and Challengescited by 3
  35. 2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applicationscited by 3
  36. 2024Revisiting Black-Box Model Ownership Verification through Information Theorycited by 2
  37. 2024From Imitation to Exploration: End-to-end Autonomous Driving based on World Modelcited by 2
  38. 2024Variable-Frequency Imitation Learning for Variable-Speed Motioncited by 2

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023End-to-end Autonomous Driving: Challenges and Frontiers
  2. 2026Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges
  3. 2023Exploring the Potential of World Models for Anomaly Detection in Autonomous Driving
  4. 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives
  5. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
  6. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  7. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions
  8. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving