mlsanityData to 14 Oct 2026

V3LMA: Visual 3D-enhanced Language Model for Autonomous Driving

arXiv 2505.00156 ↗First posted 30 Apr 2025cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Large Vision Language Models (LVLMs) have shown strong capabilities in understanding and analyzing visual scenes across various domains. However, in the context of autonomous driving, their limited comprehension of 3D environments restricts their effectiveness in achieving a complete and safe understanding of dynamic surroundings. To address this, we introduce V3LMA, a novel approach that enhances 3D scene understanding by integrating Large Language Models (LLMs) with LVLMs. V3LMA leverages textual descriptions generated from object detections and video inputs, significantly boosting performance without requiring fine-tuning. Through a dedicated preprocessing pipeline that extracts 3D object data, our method improves situational awareness and decision-making in complex traffic scenarios, achieving a score of 0.56 on the LingoQA benchmark. We further explore different fusion strategies and token combinations with the goal of advancing the interpretation of traffic scenes, ultimately enabling safer autonomous driving systems.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 38 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

25 of 38 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model13 Dec
  2. 2024Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Driving20 Nov
  3. 2024Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Driving16 Sep
  4. 2024Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Driving4 Sep
  5. 2024SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving31 Jul
  6. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  7. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving6 Dec
  8. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving3 Oct

Other references · 17

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  3. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  4. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  5. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  6. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  7. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 242
  8. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 142
  9. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  10. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  11. 2023Language Prompt for Autonomous Drivingcited by 111
  12. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
5 more
  1. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 77
  2. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  3. 2023LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoningcited by 59
  4. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotationscited by 31
  5. 2025TUMTraffic-VideoQA: A Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenescited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  2. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 10
  3. 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compressioncited by 6
  4. 2025Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perceptioncited by 2
  5. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  6. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Modelscited by 1
  7. 2026D2-V2X: Depth-Driven Cooperative V2X Reasoning for Autonomous Drivingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception
  2. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving
  3. 2026D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models
  4. 2025Scenario Understanding of Traffic Scenes Through Large Visual Language Models
  5. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
  6. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  7. 2026BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations
  8. 2024Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?