mlsanityData to 7 Oct 2026

ADriver-I: A General World Model for Autonomous Driving

arXiv 2311.13549 ↗First posted 22 Nov 2023cs.CV cs.ROcited by 102

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

Typically, autonomous driving adopts a modular design, which divides the full stack into perception, prediction, planning and control parts. Though interpretable, such modular design tends to introduce a substantial amount of redundancy. Recently, multimodal large language models (MLLM) and diffusion techniques have demonstrated their superior performance on comprehension and generation ability. In this paper, we first introduce the concept of interleaved vision-action pair, which unifies the format of visual features and control signals. Based on the vision-action pairs, we construct a general world model based on MLLM and diffusion model for autonomous driving, termed ADriver-I. It takes the vision-action pairs as inputs and autoregressively predicts the control signal of the current frame. The generated control signals together with the historical vision-action pairs are further conditioned to predict the future frames. With the predicted next frame, ADriver-I performs further control signal prediction. Such a process can be repeated infinite times, ADriver-I achieves autonomous driving in the world created by itself. Extensive experiments are conducted on nuScenes and our large-scale private datasets. ADriver-I shows impressive performance compared to several constructed baselines. We hope our ADriver-I can provide some new insights for future autonomous driving and embodied intelligence.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 54 of its 61 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

54 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  2. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep

Other references · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2017Mask R-CNNcited by 7,259
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  12. 2022Planning-oriented Autonomous Drivingcited by 1,068
  13. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  15. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  16. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  17. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  18. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  19. 2023Instruction Tuning with GPT-4cited by 676
  20. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  21. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  22. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  23. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  24. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  25. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  26. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  27. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  28. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  29. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  30. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  31. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  32. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  33. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  34. 2022Masked World Models for Visual Controlcited by 169
  35. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  36. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
  37. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  38. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 44; 2025: 33; 2026: 23.

102 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  3. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 204
  4. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  5. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  6. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  7. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  8. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  9. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  10. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 87
  11. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  12. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
38 more
  1. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  2. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  3. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  4. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPTcited by 66
  5. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  6. 20253D and 4D World Modeling: A Surveycited by 64
  7. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  8. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 54
  9. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  10. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  11. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  12. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  13. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  14. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  15. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  16. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  17. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  18. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Modelcited by 25
  19. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  20. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 24
  21. 2025A Survey of Interactive Generative Videocited by 23
  22. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  23. 2024VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Drivingcited by 22
  24. 2024RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-Worldcited by 22
  25. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real Worldcited by 22
  26. 2024HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Drivingcited by 21
  27. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  28. 2025UniFuture: A 4D Driving World Model for Future Generation and Perceptioncited by 21
  29. 2024Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?cited by 20
  30. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latentscited by 19
  31. 2025MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Drivingcited by 18
  32. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  33. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  34. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15
  35. 2024X-MOBILITY: End-To-End Generalizable Navigation via World Modelingcited by 15
  36. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignmentcited by 15
  37. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  38. 2026DriveFuture: Future-Aware Latent World Models for Autonomous Drivingcited by 15

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  2. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving
  3. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
  4. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
  5. 2026Vega: Learning to Drive with Natural Language Instructions
  6. 2025Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving
  7. 2025ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving
  8. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Model