mlsanityData to 7 Oct 2026

On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving

arXiv 2311.05332 ↗First posted 9 Nov 2023cs.CV cs.AI cs.CLcited by 79

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

The pursuit of autonomous driving technology hinges on the sophisticated integration of perception, decision-making, and control systems. Traditional approaches, both data-driven and rule-based, have been hindered by their inability to grasp the nuance of complex driving environments and the intentions of other road users. This has been a significant bottleneck, particularly in the development of common sense reasoning and nuanced scene understanding necessary for safe and reliable autonomous driving. The advent of Visual Language Models (VLM) represents a novel frontier in realizing fully autonomous vehicle driving. This report provides an exhaustive evaluation of the latest state-of-the-art VLM, GPT-4V(ision), and its application in autonomous driving scenarios. We explore the model's abilities to understand and reason about driving scenes, make decisions, and ultimately act in the capacity of a driver. Our comprehensive tests span from basic scene recognition to complex causal reasoning and real-time decision-making under varying conditions. Our findings reveal that GPT-4V demonstrates superior performance in scene understanding and causal reasoning compared to existing autonomous systems. It showcases the potential to handle out-of-distribution scenarios, recognize intentions, and make informed decisions in real driving contexts. However, challenges remain, particularly in direction discernment, traffic light recognition, vision grounding, and spatial reasoning tasks. These limitations underscore the need for further research and development. Project is now available on GitHub for interested parties to access and utilize: urlhttps://github.com/PJLab-ADG/GPT4V-AD-Exploration

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 26 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

19 of 26 references.

References · 19

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  8. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  9. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  10. 2022DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detectioncited by 270
  11. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  12. 2018Textual Explanations for Self-Driving Vehiclescited by 226
7 more
  1. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  2. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  3. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  4. 2020Uncertainty-based Traffic Accident Anticipation with Spatio-Temporal Relational Learningcited by 91
  5. 2022CitySim: A Drone-Based Vehicle Trajectory Dataset for Safety Oriented Research and Digital Twinscited by 43
  6. 2019D2-City: A Large-Scale Dashcam Video Dataset of Diverse Traffic Scenarioscited by 28
  7. 2023ADD: An Automatic Desensitization Fisheye Dataset for Autonomous Drivingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 44; 2025: 19; 2026: 6.

79 papers.

Sort:

By area:

  1. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  2. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  3. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  4. 2024Large Multimodal Agents: A Surveycited by 99
  5. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  6. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  7. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  8. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 67
  9. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  10. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  11. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  12. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
38 more
  1. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  2. 2024Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Followingcited by 41
  3. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  4. 2023Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMscited by 34
  5. 2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 33
  6. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  7. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  8. 2024VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Contextcited by 29
  9. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  10. 2024CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenariocited by 27
  11. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  12. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  13. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  14. 2024GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computingcited by 22
  15. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 21
  16. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  17. 2025When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysiscited by 20
  18. 2024Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Modelscited by 18
  19. 2024Data-Centric Evolution in Autonomous Driving: A Comprehensive Survey of Big Data System, Data Mining, and Closed-Loop Technologiescited by 17
  20. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  21. 2024Cross-Modal Consistency in Multimodal Large Language Modelscited by 17
  22. 2024GPT-4V as Traffic Assistant: An In-depth Look at Vision Language Model on Complex Traffic Eventscited by 16
  23. 2024Training-Free Open-Ended Object Detection and Segmentation via Attention as Promptscited by 15
  24. 2024PIN: Positional Insert Unlocks Object Localisation Abilities in VLMscited by 14
  25. 2025SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generationcited by 14
  26. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  27. 2023GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Predictioncited by 11
  28. 2024Malicious Path Manipulations via Exploitation of Representation Vulnerabilities of Vision-Language Navigation Systemscited by 10
  29. 2024LaVida Drive: Vision-Text Interaction VLM for Autonomous Driving with Token Selection, Recovery and Enhancementcited by 9
  30. 2024GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognitioncited by 8
  31. 2024MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planningcited by 8
  32. 2024A Survey on Human-AI Collaboration with Large Foundation Modelscited by 7
  33. 2025CoopReflect: Towards Natural Language Communication for Cooperative Autonomous Driving via Multi-Agent Learningcited by 7
  34. 2024Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Surveycited by 6
  35. 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reportscited by 6
  36. 2023An Evaluation of GPT-4V and Gemini in Online VQAcited by 5
  37. 2024Harnessing GPT-4V(ision) for Insurance: A Preliminary Explorationcited by 5
  38. 2024DRIVE: Dependable Robust Interpretable Visionary Ensemble Framework in Autonomous Drivingcited by 5

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GPT-4V Explorations: Mining Autonomous Driving
  2. 2025A Survey on Vision-Language-Action Models for Autonomous Driving
  3. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  4. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
  5. 2025LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
  6. 2025CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
  7. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
  8. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model