mlsanityData to 14 Oct 2026

Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving

arXiv 2505.08725 ↗First posted 13 May 2025cs.CVcited by 10

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

The Large Visual-Language Models (LVLMs) have significantly advanced image understanding. Their comprehension and reasoning capabilities enable promising applications in autonomous driving scenarios. However, existing research typically focuses on front-view perspectives and partial objects within scenes, struggling to achieve comprehensive scene understanding. Meanwhile, existing LVLMs suffer from the lack of mapping relationship between 2D and 3D and insufficient integration of 3D object localization and instruction understanding. To tackle these limitations, we first introduce NuInteract, a large-scale dataset with over 1.5M multi-view image language pairs spanning dense scene captions and diverse interactive tasks. Furthermore, we propose DriveMonkey, a simple yet effective framework that seamlessly integrates LVLMs with a spatial processor using a series of learnable queries. The spatial processor, designed as a plug-and-play component, can be initialized with pre-trained 3D detectors to improve 3D perception. Our experiments show that DriveMonkey outperforms general LVLMs, especially achieving a 9.86% notable improvement on the 3D visual grounding task. The dataset and code will be released at https://github.com/zc-zhao/DriveMonkey.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 85 of its 111 references that are in the map. First posted on arXiv 13 May 2025.

Builds on

85 of 111 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving27 Mar
  2. 2025Distilling Multi-modal Large Language Models for Autonomous Driving16 Jan
  3. 2025Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving15 Jan
  4. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives7 Jan
  5. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model13 Dec
  6. 2024World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving9 Dec
  7. 2024LaVida Drive: Vision-Text Interaction VLM for Autonomous Driving with Token Selection, Recovery and Enhancement20 Nov
  8. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving29 Oct
  9. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases16 Apr
  10. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  11. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models2 Jan
  12. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
5 more
  1. 2023LingoQA: Visual Question Answering for Autonomous Driving21 Dec
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  3. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving6 Dec
  4. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving3 Oct
  5. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 68

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Segment Anythingcited by 8,234
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2025Qwen2.5-VL Technical Reportcited by 5,587
  12. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  7. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  10. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  13. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  14. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  15. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  16. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  17. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
  18. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  19. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  20. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 572
  21. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  22. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  23. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  24. 2024InternLM2 Technical Reportcited by 397
  25. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  26. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 352
  27. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  28. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  29. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  30. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  31. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  32. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  33. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 178
  34. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 142
  35. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  36. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  37. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 119
  38. 2023Language Prompt for Autonomous Drivingcited by 111

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  2. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  3. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  4. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  5. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 10
  6. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 9
  7. 2026SparseOccVLA: Bridging Occupancy and Vision-Language Models via Sparse Queries for Unified 4D Scene Understanding and Planningcited by 4
  8. 2026HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generationcited by 3
  9. 2026GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generationcited by 0
  10. 2026Beyond Attention Imbalance: Mitigating Hallucinations via Spectral Surgerycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025V3LMA: Visual 3D-enhanced Language Model for Autonomous Driving
  2. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
  3. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models
  4. 2025LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
  5. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
  6. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Driving
  7. 2025NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Driving
  8. 2026ScenePilot-4K: A Large-Scale First-Person Dataset and Benchmark for Vision-Language Models in Autonomous Driving