mlsanityData to 7 Oct 2026

DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers

arXiv 2412.18607 ↗First posted 24 Dec 2024cs.CVcited by 96

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

World model-based searching and planning are widely recognized as a promising path toward human-level physical intelligence. However, current driving world models primarily rely on video diffusion models, which specialize in visual generation but lack the flexibility to incorporate other modalities like action. In contrast, autoregressive transformers have demonstrated exceptional capability in modeling multimodal data. Our work aims to unify both driving model simulation and trajectory planning into a single sequence modeling problem. We introduce a multimodal driving language based on interleaved image and action tokens, and develop DrivingGPT to learn joint world modeling and planning through standard next-token prediction. Our DrivingGPT demonstrates strong performance in both action-conditioned video generation and end-to-end planning, outperforming strong baselines on large-scale nuPlan and NAVSIM benchmarks.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 70 of its 82 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

70 of 82 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model14 Oct
  2. 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model14 Oct
  3. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving5 Sep
  4. 2024Enhancing End-to-End Autonomous Driving with Latent World Model12 Jun
  5. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  6. 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
  7. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  8. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  9. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
  10. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion2 Nov
  11. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research12 Oct
  12. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
3 more
  1. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  2. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun
  3. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
38 more
  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2018World Modelscited by 1,444
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  9. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  10. 2022Planning-oriented Autonomous Drivingcited by 1,068
  11. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  12. 2016End to End Learning for Self-Driving Carscited by 1,000
  13. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  14. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  15. 2016Pixel Recurrent Neural Networkscited by 814
  16. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  17. 2018Image Transformercited by 770
  18. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  19. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  20. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  21. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  22. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  23. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  24. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  25. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  26. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  27. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  28. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  29. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  30. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  31. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  32. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  33. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  34. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  35. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  36. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  37. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  38. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 40; 2026: 56.

96 papers.

Sort:

By area:

  1. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  2. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  3. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  4. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  5. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  6. 2025ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Drivingcited by 66
  7. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  8. 20253D and 4D World Modeling: A Surveycited by 64
  9. 2025A Survey of World Models for Autonomous Drivingcited by 60
  10. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  11. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  12. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
38 more
  1. 2025DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Drivingcited by 39
  2. 2026Driving on Registerscited by 38
  3. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  4. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  5. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  6. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  7. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  8. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  9. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 24
  10. 2025Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modelingcited by 22
  11. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real Worldcited by 22
  12. 2025UniFuture: A 4D Driving World Model for Future Generation and Perceptioncited by 21
  13. 2025DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulationcited by 21
  14. 2026MobileDreamer: Generative Sketch World Model for GUI Agentcited by 21
  15. 2026DriveVA: Video Action Models are Zero-Shot Driverscited by 19
  16. 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planningcited by 17
  17. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignmentcited by 15
  18. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  19. 2025WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Drivingcited by 15
  20. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Modelscited by 13
  21. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
  22. 2026EponaV2: Driving World Model with Comprehensive Future Reasoningcited by 12
  23. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  24. 2025GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generationcited by 10
  25. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 10
  26. 2026CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Drivingcited by 10
  27. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 10
  28. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 10
  29. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  30. 2025MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Casescited by 9
  31. 2026FLARE: Learning Future-Aware Latent Representations from Vision-Language Models for Autonomous Drivingcited by 9
  32. 2026Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Drivingcited by 9
  33. 2026CLOVER: Closed-Loop Value Estimation and Ranking for End-to-End Autonomous Driving Planningcited by 9
  34. 2025Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenescited by 7
  35. 2026Test-Time Trajectory Optimization for Autonomous Drivingcited by 7
  36. 2026Beyond Imitation: Learning Safe End-to-End Autonomous Driving from Hard Negativescited by 6
  37. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 6
  38. 2026GraphWorld: Long-Horizon Planning with World Models for End-to-End Autonomous Drivingcited by 6

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
  2. 2025ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos
  3. 2026Drive-HWM: Hierarchical World Models for Dynamic-Latent Guided Autonomous Driving
  4. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving
  5. 2024DriveGPT: Scaling Autoregressive Behavior Models for Driving
  6. 2026Multimodal Action Diffusion for Robust End-to-End Autonomous Driving
  7. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving
  8. 2026SimWAM: A Simple World Action Model for End-to-End Autonomous Driving