mlsanityData to 7 Oct 2026

Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models

arXiv 2401.00988 ↗First posted 2 Jan 2024cs.CVcited by 75

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

The rise of multimodal large language models (MLLMs) has spurred interest in language-based driving tasks. However, existing research typically focuses on limited tasks and often omits key multi-view and temporal information which is crucial for robust autonomous driving. To bridge these gaps, we introduce NuInstruct, a novel dataset with 91K multi-view video-QA pairs across 17 subtasks, where each task demands holistic information (e.g., temporal, multi-view, and spatial), significantly elevating the challenge level. To obtain NuInstruct, we propose a novel SQL-based method to generate instruction-response pairs automatically, which is inspired by the driving logical progression of humans. We further present BEV-InMLLM, an end-to-end method for efficiently deriving instruction-aware Bird's-Eye-View (BEV) features, language-aligned for large language models. BEV-InMLLM integrates multi-view, spatial awareness, and temporal semantics to enhance MLLMs' capabilities on NuInstruct tasks. Moreover, our proposed BEV injection module is a plug-and-play method for existing MLLMs. Our experiments on NuInstruct demonstrate that BEV-InMLLM significantly outperforms existing MLLMs, e.g. around 9% improvement on various tasks. We plan to release our NuInstruct for future research development.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 56 references that are in the map. First posted on arXiv 2 Jan 2024.

Builds on

43 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving3 Oct
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
29 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  4. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  5. 2022Planning-oriented Autonomous Drivingcited by 1,068
  6. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  8. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  9. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  10. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  11. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  12. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  13. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  14. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  15. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  16. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  17. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  18. 2022Vision-Centric BEV Perception: A Surveycited by 126
  19. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  20. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  21. 2023Language Prompt for Autonomous Drivingcited by 110
  22. 2023Referring Multi-Object Trackingcited by 91
  23. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  24. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  25. 2018Object Referring in Videos with Language and Human Gazecited by 40
  26. 2022Relation Matters: Foreground-aware Graph-based Relational Reasoning for Domain Adaptive Object Detectioncited by 17
  27. 2023UniM2AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Drivingcited by 12
  28. 2023Boosting Weakly-Supervised Temporal Action Localization with Text Informationcited by 8
  29. 2023EtC: Temporal Boundary Expand then Clarify for Weakly Supervised Video Grounding with Multimodal Large Language Modelcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 42; 2026: 21.

75 papers.

Sort:

By area:

  1. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  2. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  3. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  4. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  5. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  6. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  7. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 55
  8. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  9. 2024Graphic Design with Large Multimodal Modelcited by 38
  10. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  11. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
38 more
  1. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understandingcited by 35
  2. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  3. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  4. 2025Token Activation Map to Visually Explain Multimodal LLMscited by 30
  5. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  6. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  7. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 26
  8. 2025BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Drivingcited by 26
  9. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  10. 2025S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representationcited by 18
  11. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  12. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 17
  13. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15
  14. 2025PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learningcited by 15
  15. 2025RefAV: Towards Planning-Centric Scenario Miningcited by 13
  16. 2025DynRsl-VLM: Enhancing Autonomous Driving Perception with Dynamic Resolution Vision-Language Modelscited by 12
  17. 2025NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Drivingcited by 12
  18. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  19. 2024MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Imagescited by 9
  20. 2024MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarioscited by 8
  21. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 8
  22. 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Drivingcited by 8
  23. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 8
  24. 2025Tracking Meets Large Multimodal Models for Driving Scenario Understandingcited by 7
  25. 2024DualAD: Dual-Layer Planning for Reasoning in Autonomous Drivingcited by 6
  26. 2025EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Abilitycited by 6
  27. 2025NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Drivingcited by 6
  28. 2026Vega: Learning to Drive with Natural Language Instructionscited by 6
  29. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  30. 2025A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Drivingcited by 5
  31. 2025A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospectscited by 5
  32. 2025RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Casecited by 4
  33. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Drivingcited by 3
  34. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  35. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  36. 2025AutoDrive-QA: A Multiple-Choice Benchmark for Vision-Language Evaluation in Urban Autonomous Drivingcited by 2
  37. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  38. 2025Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perceptioncited by 2

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations
  2. 2025VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion
  3. 2024World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving
  4. 2026V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
  5. 2026MoRAL: Sensor-Grounded BEV Reasoning for Compact VLMs toward Edge-Oriented Autonomous Driving
  6. 2026SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving
  7. 2025A Survey on Vision-Language-Action Models for Autonomous Driving
  8. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model