mlsanityData to 7 Oct 2026

DriveLM: Driving with Graph Visual Question Answering

arXiv 2312.14150 ↗First posted 21 Dec 2023cs.CVcited by 527

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

We study how vision-language models (VLMs) trained on web-scale data can be integrated into end-to-end driving systems to boost generalization and enable interactivity with human users. While recent approaches adapt VLMs to driving via single-round visual question answering (VQA), human drivers reason about decisions in multiple steps. Starting from the localization of key objects, humans estimate object interactions before taking actions. The key insight is that with our proposed task, Graph VQA, where we model graph-structured reasoning through perception, prediction and planning question-answer pairs, we obtain a suitable proxy task to mimic the human reasoning process. We instantiate datasets (DriveLM-Data) built upon nuScenes and CARLA, and propose a VLM-based baseline approach (DriveLM-Agent) for jointly performing Graph VQA and end-to-end driving. The experiments demonstrate that Graph VQA provides a simple, principled framework for reasoning about a driving scene, and DriveLM-Data provides a challenging benchmark for this task. Our DriveLM-Agent baseline performs end-to-end autonomous driving competitively in comparison to state-of-the-art driving-specific architectures. Notably, its benefits are pronounced when it is evaluated zero-shot on unseen objects or sensor configurations. We hope this work can be the starting point to shed new light on how to apply VLMs for autonomous driving. To facilitate future research, all code, data, and models are available to the public.

Key facts. Cited by 527 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 76 of its 101 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

76 of 101 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  2. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving3 Oct
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 73

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
38 more
  1. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2022Planning-oriented Autonomous Drivingcited by 1,068
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  13. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  14. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  15. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  16. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  17. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  18. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  19. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  20. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  21. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  22. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  23. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  24. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  25. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  26. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  27. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  28. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  29. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  30. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 214
  31. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  32. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  33. 2023Language-Driven Representation Learning for Roboticscited by 163
  34. 2023A Language Agent for Autonomous Drivingcited by 161
  35. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  36. 2022Learning from All Vehiclescited by 135
  37. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 132
  38. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 97; 2025: 188; 2026: 240.

527 papers.

Sort:

By area:

  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  2. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  3. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  4. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  6. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  7. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  8. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  9. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  10. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  11. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  12. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
38 more
  1. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  2. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  3. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  4. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  5. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  6. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  7. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  8. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  9. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  10. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  11. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
  12. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  13. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  14. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  15. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  16. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  17. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  18. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  19. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  20. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  21. 2024Embodied Understanding of Driving Scenarioscited by 63
  22. 2025Generalized Trajectory Scoring for End-to-end Multimodal Planningcited by 63
  23. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  24. 2023LaneSegNet: Map Learning with Lane Segment Perception for Autonomous Drivingcited by 57
  25. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  26. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  27. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  28. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 55
  29. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Drivingcited by 52
  30. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Modelscited by 52
  31. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  32. 2025FineVision: Open Data Is All You Needcited by 51
  33. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  34. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  35. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  36. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  37. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  38. 2025Centaur: Robust End-to-End Autonomous Driving with Test-Time Trainingcited by 41

50 most cited of 527 All 527 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Driving
  2. 2025STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes
  3. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving
  4. 2025Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios
  5. 2025Hierarchical Question-Answering for Driving Scene Understanding Using Vision-Language Models
  6. 2023LingoQA: Visual Question Answering for Autonomous Driving
  7. 2025RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System
  8. 2025Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving