mlsanityData to 7 Oct 2026

Dolphins: Multimodal Language Model for Driving

arXiv 2312.00438 ↗First posted 1 Dec 2023cs.CVcited by 124

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

The quest for fully autonomous vehicles (AVs) capable of navigating complex real-world scenarios with human-like understanding and responsiveness. In this paper, we introduce Dolphins, a novel vision-language model architected to imbibe human-like abilities as a conversational driving assistant. Dolphins is adept at processing multimodal inputs comprising video (or image) data, text instructions, and historical control signals to generate informed outputs corresponding to the provided instructions. Building upon the open-sourced pretrained Vision-Language Model, OpenFlamingo, we first enhance Dolphins's reasoning capabilities through an innovative Grounded Chain of Thought (GCoT) process. Then we tailored Dolphins to the driving domain by constructing driving-specific instruction data and conducting instruction tuning. Through the utilization of the BDD-X dataset, we designed and consolidated four distinct AV tasks into Dolphins to foster a holistic understanding of intricate driving scenarios. As a result, the distinctive features of Dolphins are characterized into two dimensions: (1) the ability to provide a comprehensive understanding of complex and long-tailed open-world driving scenarios and solve a spectrum of AV tasks, and (2) the emergence of human-like capabilities including gradient-free instant adaptation via in-context learning and error recovery via reflection.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 75 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

57 of 75 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  13. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  14. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  15. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  16. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  17. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  18. 2021MetaICL: Learning to Learn In Contextcited by 450
  19. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  20. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  21. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  22. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  23. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  24. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  25. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  26. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  27. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  28. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  29. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  30. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  31. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  32. 2021Meta-learning via Language Model In-context Tuningcited by 163
  33. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  34. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  35. 2020A Survey of End-to-End Driving: Architectures and Training Methodscited by 123
  36. 2023Language Prompt for Autonomous Drivingcited by 110
  37. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  38. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 36; 2025: 53; 2026: 33.

124 papers.

Sort:

By area:

  1. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  2. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  3. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  4. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  5. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  6. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  7. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 133
  8. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  9. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  10. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  11. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  12. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
38 more
  1. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 55
  2. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  3. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
  4. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  5. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  6. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  7. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  8. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  9. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  10. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  11. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  13. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  14. 2024Language-Image Models with 3D Understandingcited by 31
  15. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  16. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  17. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  18. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  19. 2024CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenariocited by 27
  20. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  21. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  22. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 23
  23. 2025Black-Box Adversarial Attack on Vision Language Models for Autonomous Drivingcited by 23
  24. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 23
  25. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  26. 2024VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Drivingcited by 22
  27. 2024The Evolution of Multimodal Model Architecturescited by 21
  28. 2025VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understandingcited by 20
  29. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  30. 2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agentscited by 19
  31. 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 19
  32. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  33. 2025S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representationcited by 18
  34. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  35. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 17
  36. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15
  37. 2024On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validationcited by 15
  38. 2025RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narrativescited by 15

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MoRAL: Sensor-Grounded BEV Reasoning for Compact VLMs toward Edge-Oriented Autonomous Driving
  2. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models
  3. 2023A Language Agent for Autonomous Driving
  4. 2025BeLLA: End-to-End Birds Eye View Large Language Assistant for Autonomous Driving
  5. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving
  6. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
  7. 2025A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Driving
  8. 2024Probing Multimodal LLMs as World Models for Driving