mlsanityData to 7 Oct 2026

DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving

arXiv 2405.04390 ↗First posted 7 May 2024cs.CVcited by 89

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

Vision-centric autonomous driving has recently raised wide attention due to its lower cost. Pre-training is essential for extracting a universal representation. However, current vision-centric pre-training typically relies on either 2D or 3D pre-text tasks, overlooking the temporal characteristics of autonomous driving as a 4D scene understanding task. In this paper, we address this challenge by introducing a world model-based autonomous driving 4D representation learning framework, dubbed DriveWorld, which is capable of pre-training from multi-camera driving videos in a spatio-temporal fashion. Specifically, we propose a Memory State-Space Model for spatio-temporal modelling, which consists of a Dynamic Memory Bank module for learning temporal-aware latent dynamics to predict future changes and a Static Scene Propagation module for learning spatial-aware latent statics to offer comprehensive scene contexts. We additionally introduce a Task Prompt to decouple task-aware features for various downstream tasks. The experiments demonstrate that DriveWorld delivers promising results on various autonomous driving tasks. When pre-trained with the OpenScene dataset, DriveWorld achieves a 7.5% increase in mAP for 3D object detection, a 3.0% increase in IoU for online mapping, a 5.0% increase in AMOTA for multi-object tracking, a 0.1m decrease in minADE for motion forecasting, a 3.0% increase in IoU for occupancy prediction, and a 0.34m reduction in average L2 error for planning.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 24 of its 99 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

24 of 99 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion2 Nov
  2. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  3. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  4. 2023Exploring the Potential of World Models for Anomaly Detection in Autonomous Driving10 Aug
  5. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model8 Oct

Other references · 19

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2018World Modelscited by 1,444
  3. 2023Mastering Diverse Domains through World Modelscited by 1,193
  4. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  5. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 340
  6. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  7. 2023Structured World Models from Human Videoscited by 144
  8. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  9. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  10. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  11. 2022STS: Surround-view Temporal Stereo for Multi-view 3D Detectioncited by 53
  12. 2023LiDAR-based 4D Occupancy Completion and Forecastingcited by 23
7 more
  1. 2023Self-supervised Learning for Pre-Training 3D Point Clouds: A Surveycited by 21
  2. 2023GeoMIM: Towards Better 3D Knowledge Transfer via Masked Image Modeling for Multi-view 3D Understandingcited by 17
  3. 2021Deep Learning for Multi-View Stereo via Plane Sweep: A Surveycited by 16
  4. 2022CO^3: Cooperative Unsupervised 3D Representation Learning for Autonomous Drivingcited by 15
  5. 2023BEVGPT: Generative Pre-trained Large Model for Autonomous Driving Prediction, Decision-Making, and Planningcited by 15
  6. 2023TSP-Transformer: Task-Specific Prompts Boosted Transformer for Holistic Scene Understandingcited by 12
  7. 2023SPOT: Scalable 3D Pre-training via Occupancy Prediction for Learning Transferable 3D Representationscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 38; 2026: 38.

89 papers.

Sort:

By area:

  1. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  2. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  3. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  4. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  5. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  6. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  7. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  8. 20253D and 4D World Modeling: A Surveycited by 64
  9. 2025A Survey of World Models for Autonomous Drivingcited by 60
  10. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  11. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  12. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 50
38 more
  1. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  2. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  4. 2024A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspectivecited by 37
  5. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  6. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  7. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  8. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  9. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  10. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Drivingcited by 27
  11. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
  12. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  13. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latentscited by 19
  14. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  15. 2025QuadricFormer: Scene as Superquadrics for 3D Semantic Occupancy Predictioncited by 19
  16. 2025WorldRFT: Latent World Model Planning with Reinforcement Fine-Tuning for Autonomous Drivingcited by 19
  17. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  18. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignmentcited by 15
  19. 2026DriveFuture: Future-Aware Latent World Models for Autonomous Drivingcited by 15
  20. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  21. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
  22. 2025ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Drivingcited by 12
  23. 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Modelscited by 12
  24. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  25. 2025Four Principles for Physically Interpretable World Modelscited by 10
  26. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  27. 2026CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Drivingcited by 10
  28. 2024World Models: The Safety Perspectivecited by 9
  29. 2025I2-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecastingcited by 8
  30. 2024Autonomous Driving in Unstructured Environments: How Far Have We Come?cited by 7
  31. 2025Flux4D: Flow-based Unsupervised 4D Reconstructioncited by 7
  32. 2026SafeDrive: Fine-Grained Safety Reasoning for End-to-End Driving in a Sparse Worldcited by 7
  33. 2025Fully Unified Motion Planning for End-to-End Autonomous Drivingcited by 6
  34. 2026DreamWorld: Unified World Modeling in Video Generationcited by 6
  35. 2026Vega: Learning to Drive with Natural Language Instructionscited by 6
  36. 2025INSIGHT: Enhancing Autonomous Driving Safety through Vision-Language Models on Context-Aware Hazard Detection and Edge Case Evaluationcited by 5
  37. 2025Enhancing Physical Consistency in Lightweight World Modelscited by 5
  38. 2025Spatial Retrieval Augmented Autonomous Drivingcited by 5

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation
  2. 2026ScenePilot-4K: A Large-Scale First-Person Dataset and Benchmark for Vision-Language Models in Autonomous Driving
  3. 2024DreamDrive: Generative 4D Scene Modeling from Street View Images
  4. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
  5. 2025Tracking Meets Large Multimodal Models for Driving Scenario Understanding
  6. 2026ResWorld: Temporal Residual World Model for End-to-End Autonomous Driving
  7. 2026DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving
  8. 2026UniDWM: Towards a Unified Driving World Model via Multifaceted Representation Learning