mlsanityData to 7 Oct 2026

DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

arXiv 2402.12289 ↗First posted 19 Feb 2024cs.CVcited by 482

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

A primary hurdle of autonomous driving in urban environments is understanding complex and long-tail scenarios, such as challenging road conditions and delicate human behaviors. We introduce DriveVLM, an autonomous driving system leveraging Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities. DriveVLM integrates a unique combination of reasoning modules for scene description, scene analysis, and hierarchical planning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, we propose DriveVLM-Dual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driving pipeline. Experiments on both the nuScenes dataset and our SUP-AD dataset demonstrate the efficacy of DriveVLM and DriveVLM-Dual in handling complex and unpredictable driving conditions. Finally, we deploy the DriveVLM-Dual on a production vehicle, verifying it is effective in real-world autonomous driving environments.

Key facts. Cited by 482 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 78 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

59 of 78 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 57

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  3. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  4. 2022Planning-oriented Autonomous Drivingcited by 1,068
  5. 2016Modeling Context in Referring Expressionscited by 1,054
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  7. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  11. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  12. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  13. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  14. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  15. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  16. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  17. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  18. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  19. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  20. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  21. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  22. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  23. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  24. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  25. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  26. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  27. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  28. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  29. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  30. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  31. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  32. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  33. 2021Learning to drive from a world on railscited by 114
  34. 2023Language Prompt for Autonomous Drivingcited by 110
  35. 2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 106
  36. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  37. 2023Referring Multi-Object Trackingcited by 91
  38. 2020Explainable Object-induced Action Decision for Autonomous Vehiclescited by 88

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 70; 2025: 215; 2026: 197.

482 papers.

Sort:

By area:

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  2. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  3. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  4. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  5. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  6. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  7. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  8. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  9. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  10. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  11. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  12. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
38 more
  1. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  2. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  3. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  4. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  5. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  6. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  7. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
  8. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  9. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  10. 2025Visual Spatial Tuningcited by 85
  11. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 83
  12. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  13. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  14. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 71
  15. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
  16. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  17. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  18. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  19. 2025WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarioscited by 60
  20. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  21. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 55
  22. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  23. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  24. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  25. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
  26. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  27. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  28. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  29. 2024RenderWorld: World Model with Self-Supervised 3D Labelcited by 44
  30. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  31. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  32. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learningcited by 38
  33. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  34. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  35. 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Drivingcited by 35
  36. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Drivingcited by 34
  37. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  38. 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33

50 most cited of 482 All 482 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VLMPlanner: Integrating Visual Language Models with Motion Planning
  2. 2026WorldVLM: Combining World Model Forecasting and Vision-Language Reasoning
  3. 2025LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
  4. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  5. 2024VLP: Vision Language Planning for Autonomous Driving
  6. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving
  7. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios
  8. 2025CARScenes: Semantic VLM Dataset for Safe Autonomous Driving