mlsanityData to 7 Oct 2026

Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving

arXiv 2403.19838 ↗First posted 28 Mar 2024cs.CV cs.AIcited by 52

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) and Multi-Modal Language models (MMLMs) have become prominent in autonomous driving research, as these models can provide interpretable textual reasoning and responses for end-to-end autonomous driving safety tasks using traffic scene images and other data modalities. However, current approaches to these systems use expensive large language model (LLM) backbones and image encoders, making such systems unsuitable for real-time autonomous driving systems where tight memory constraints exist and fast inference time is necessary. To address these previous issues, we develop EM-VLM4AD, an efficient, lightweight, multi-frame vision language model which performs Visual Question Answering for autonomous driving. In comparison to previous approaches, EM-VLM4AD requires at least 10 times less memory and floating point operations, while also achieving higher CIDEr and ROUGE-L scores than the existing baseline on the DriveLM dataset. EM-VLM4AD also exhibits the ability to extract relevant information from traffic views related to prompts and can answer questions for various autonomous driving subtasks. We release our code to train and evaluate our model at https://github.com/akshaygopalkr/EM-VLM4AD.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 37 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

28 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
13 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  3. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  4. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  5. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  6. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  7. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  8. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  9. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  10. 2024Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Setscited by 25
  11. 2024The Why, When, and How to Use Active Learning in Large-Data-Driven 3D Object Detection for Safe Autonomous Driving: An Empirical Explorationcited by 17
  12. 2023Robust Traffic Light Detection Using Salience-Sensitive Loss: Computational Framework and Evaluationscited by 11
  13. 2023MIVC: Multiple Instance Visual Component for Visual-Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 28; 2026: 13.

52 papers.

Sort:

By area:

  1. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  2. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  3. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  4. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Drivingcited by 34
  5. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  6. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  7. 2025VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understandingcited by 20
  8. 2025RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narrativescited by 15
  9. 2025SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generationcited by 14
  10. 2024MiniDrive: More Efficient Vision-Language Models with Multi-Level 2D Features as Text Tokens for Autonomous Drivingcited by 13
  11. 2025DynRsl-VLM: Enhancing Autonomous Driving Perception with Dynamic Resolution Vision-Language Modelscited by 12
  12. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
38 more
  1. 2024Right this way: Can VLMs Guide Us to See More to Answer Questions?cited by 10
  2. 2024LaVida Drive: Vision-Text Interaction VLM for Autonomous Driving with Token Selection, Recovery and Enhancementcited by 9
  3. 2025ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Modelscited by 8
  4. 2025Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Drivingcited by 8
  5. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  6. 2025VLM-Assisted Continual learning for Visual Question Answering in Self-Drivingcited by 6
  7. 2025LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Drivingcited by 6
  8. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  9. 2025TS-VLM: Text-Guided SoftSort Pooling for Vision-Language Models in Multi-View Driving Reasoningcited by 5
  10. 2025Automated Vehicles Should be Connected with Natural Languagecited by 5
  11. 2026Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Modelscited by 5
  12. 2024PG-Attack: A Precision-Guided Adversarial Attack Framework Against Vision Foundation Models for Autonomous Drivingcited by 4
  13. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4
  14. 2025DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Modelscited by 4
  15. 2025LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Drivingcited by 4
  16. 2024From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensingcited by 3
  17. 2025Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactionscited by 3
  18. 2025RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation Systemcited by 3
  19. 2025MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusioncited by 3
  20. 2024Perception Without Vision for Trajectory Prediction: Ego Vehicle Dynamics as Scene Representation for Efficient Active Learning in Autonomous Drivingcited by 2
  21. 2024Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safetycited by 2
  22. 2025Urban Computing in the Era of Large Language Modelscited by 2
  23. 2025Image Segmentation with Large Language Models: A Survey with Perspectives for Intelligent Transportation Systemscited by 2
  24. 2025DriveMRP: Enhancing Vision-Language Models with Synthetic Motion Data for Motion Risk Predictioncited by 2
  25. 2025Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perceptioncited by 2
  26. 2026Drive-P2D: A Progressive Perception-to-Decision Benchmark for VLMs in Autonomous Drivingcited by 2
  27. 2026Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoningcited by 2
  28. 2026How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Studycited by 2
  29. 2025TinyDrive: Multiscale Visual Question Answering with Selective Token Routing for Autonomous Drivingcited by 1
  30. 2025DriveBLIP2: Attention-Guided Explanation Generation for Complex Driving Scenarioscited by 1
  31. 2026Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planningcited by 1
  32. 2026Multimodal Backdoor Attack on VLMs for Autonomous Driving via Graffiti and Cross-Lingual Triggerscited by 1
  33. 2026DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Modelscited by 1
  34. 2025BeLLA: End-to-End Birds Eye View Large Language Assistant for Autonomous Drivingcited by 0
  35. 2026Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Makingcited by 0
  36. 2026Beyond MACs: Hardware Efficient Architecture Design for Vision Backbonescited by 0
  37. 2026GuardAD: Safeguarding Autonomous Driving MLLMs via Markovian Safety Logiccited by 0
  38. 2026Overthink-Triggered Slowdown Attacks on LVLM-Based Robotic Systemscited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression
  2. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving
  3. 2024SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving
  4. 2026Reducing Text Bias in Synthetically Generated MCQAs for VLMs in Autonomous Driving
  5. 2025Hierarchical Question-Answering for Driving Scene Understanding Using Vision-Language Models
  6. 2025VLMs Guided Interpretable Decision Making for Autonomous Driving
  7. 2023LingoQA: Visual Question Answering for Autonomous Driving
  8. 2025Unlock the Power of Unlabeled Data in Language Driving Model