mlsanityData to 7 Oct 2026

Doe-1: Closed-Loop Autonomous Driving with Large World Model

arXiv 2412.09627 ↗First posted 12 Dec 2024cs.CV cs.AI cs.LGcited by 51

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

End-to-end autonomous driving has received increasing attention due to its potential to learn from large amounts of data. However, most existing methods are still open-loop and suffer from weak scalability, lack of high-order interactions, and inefficient decision-making. In this paper, we explore a closed-loop framework for autonomous driving and propose a large Driving wOrld modEl (Doe-1) for unified perception, prediction, and planning. We formulate autonomous driving as a next-token generation problem and use multi-modal tokens to accomplish different tasks. Specifically, we use free-form texts (i.e., scene descriptions) for perception and generate future predictions directly in the RGB space with image tokens. For planning, we employ a position-aware tokenizer to effectively encode action into discrete tokens. We train a multi-modal transformer to autoregressively generate perception, prediction, and planning tokens in an end-to-end and unified manner. Experiments on the widely used nuScenes dataset demonstrate the effectiveness of Doe-1 in various tasks including visual question-answering, action-conditioned video generation, and motion planning. Code: https://github.com/wzzheng/Doe.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 51 of its 101 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

51 of 101 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving26 Aug
  2. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation3 Jun
  3. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  4. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations20 Nov
  5. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  6. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2017Decoupled Weight Decay Regularizationcited by 1,722
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2018World Modelscited by 1,444
33 more
  1. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  2. 2016End to End Learning for Self-Driving Carscited by 1,000
  3. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  4. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  5. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  6. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  7. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  8. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  9. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  10. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  11. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  12. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  13. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  14. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  15. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 155
  16. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  17. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  18. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  19. 2024GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Predictioncited by 83
  20. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  21. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  22. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  23. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  24. 2023LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoningcited by 59
  25. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  26. 2023PointOcc: Cylindrical Tri-Perspective View for Point-based 3D Semantic Occupancy Predictioncited by 51
  27. 2024SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Drivingcited by 46
  28. 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 43
  29. 2024Driv3R: Learning Dense 4D Reconstruction for Autonomous Drivingcited by 28
  30. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 26
  31. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15
  32. 2024Instruct Large Language Models to Drive like Humanscited by 11
  33. 2024GPD-1: Generative Pre-training for Drivingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 28.

51 papers.

Sort:

By area:

  1. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  2. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  3. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  4. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  5. 20253D and 4D World Modeling: A Surveycited by 64
  6. 2025A Survey of World Models for Autonomous Drivingcited by 60
  7. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  8. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  9. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  10. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  11. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  12. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
38 more
  1. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  2. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  3. 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Drivingcited by 26
  4. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  5. 2025DVGT: Driving Visual Geometry Transformercited by 20
  6. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Drivingcited by 20
  7. 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Drivingcited by 20
  8. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  9. 2026DriveVA: Video Action Models are Zero-Shot Driverscited by 19
  10. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  11. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 12
  12. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  13. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  14. 2026MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Drivingcited by 10
  15. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 10
  16. 2025LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Modelcited by 9
  17. 2026Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Drivingcited by 9
  18. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  19. 2026Vega: Learning to Drive with Natural Language Instructionscited by 6
  20. 2025The Safety Challenge of World Models for Embodied AI Agents: A Reviewcited by 5
  21. 2026Human Cognition in Machines: A Unified Perspective of World Modelscited by 5
  22. 2026UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generationcited by 5
  23. 2026SimWAM: A Simple World Action Model for End-to-End Autonomous Drivingcited by 5
  24. 2025Terra: Explorable Native 3D World Model with Point Latentscited by 4
  25. 2026HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generationcited by 3
  26. 2026WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMscited by 2
  27. 2026GeoWAM: Visual Geometry World Action Models for Autonomous Drivingcited by 2
  28. 2026HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Drivingcited by 1
  29. 2026Adaptive-WAM: Quality-Guided Early-Exit Planning from Intermediate Video-Diffusion Featurescited by 1
  30. 2026SV-WAM: An Efficient Surround-View World-Action Model for End-to-End Autonomous Drivingcited by 1
  31. 2026World-Action Models for Robot Learning and Control: A Surveycited by 1
  32. 2025Enhancing End-to-End Autonomous Driving with Risk Semantic Distillaion from VLMcited by 0
  33. 2026DISK: Dynamic Inference SKipping for World Modelscited by 0
  34. 2026DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understandingcited by 0
  35. 2026LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Modelcited by 0
  36. 2026TPS-Drive: Task-Guided Representation Purification for VLM-based Autonomous Drivingcited by 0
  37. 2026SUV: Future Scene Understanding as Video Generation for End-to-End Drivingcited by 0
  38. 2026From Generation to Simulation: How Far Are World Models from Being True Simulators?cited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ADriver-I: A General World Model for Autonomous Driving
  2. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
  3. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models
  4. 2026FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving
  5. 2025ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving
  6. 2026LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
  7. 2025Less is More: Lean yet Powerful Vision-Language Model for Autonomous Driving
  8. 2025X-Driver: Explainable Autonomous Driving with Vision-Language Models