mlsanityData to 14 Oct 2026

Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving

arXiv 2310.02251 ↗First posted 3 Oct 2023cs.CV cs.ROcited by 89

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Talk2BEV is a large vision-language model (LVLM) interface for bird's-eye view (BEV) maps in autonomous driving contexts. While existing perception systems for autonomous driving scenarios have largely focused on a pre-defined (closed) set of object categories and driving scenarios, Talk2BEV blends recent advances in general-purpose language and vision models with BEV-structured map representations, eliminating the need for task-specific models. This enables a single system to cater to a variety of autonomous driving tasks encompassing visual and spatial reasoning, predicting the intents of traffic actors, and decision-making based on visual cues. We extensively evaluate Talk2BEV on a large number of scene understanding tasks that rely on both the ability to interpret free-form natural language queries, and in grounding these queries to the visual context embedded into the language-enhanced BEV map. To enable further research in LVLMs for autonomous driving scenarios, we develop and release Talk2BEV-Bench, a benchmark encompassing 1000 human-annotated BEV scenarios, with more than 20,000 questions and ground-truth responses from the NuScenes dataset.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 49 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

38 of 49 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
26 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  5. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  7. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  8. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  9. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  10. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  11. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  12. 2023Fast Segment Anythingcited by 290
  13. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  14. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  15. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  16. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  17. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 151
  18. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 142
  19. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 121
  20. 2023Language Prompt for Autonomous Drivingcited by 111
  21. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  22. 2023Referring Multi-Object Trackingcited by 93
  23. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  24. 2020MonoLayout: Amodal scene layout from a single imagecited by 49
  25. 2018Object Referring in Videos with Language and Human Gazecited by 40
  26. 2021Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulationcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 23; 2025: 40; 2026: 17.

89 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  2. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  3. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  4. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  5. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  6. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 123
  7. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 86
  8. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 83
  9. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 76
  10. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  11. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 66
  12. 2024Embodied Understanding of Driving Scenarioscited by 63
38 more
  1. 2024A Survey for Foundation Models in Autonomous Drivingcited by 52
  2. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  4. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  5. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  6. 2024LLM-Assisted Light: Leveraging Large Language Model Capabilities for Human-Mimetic Traffic Signal Control in Complex Urban Environmentscited by 31
  7. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  8. 2025Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Modelscited by 28
  9. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 27
  10. 2025BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Drivingcited by 26
  11. 2025DiffAD: A Unified Diffusion Modeling Approach for Autonomous Drivingcited by 26
  12. 2024LG-Traj: LLM Guided Pedestrian Trajectory Predictioncited by 25
  13. 2025Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learningcited by 25
  14. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  15. 2025A Survey on Efficient Vision-Language Modelscited by 19
  16. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  17. 2025Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directionscited by 14
  18. 2025DynRsl-VLM: Enhancing Autonomous Driving Perception with Dynamic Resolution Vision-Language Modelscited by 12
  19. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  20. 2024World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Drivingcited by 9
  21. 2024Generating and Evolving Reward Functions for Highway Driving with Large Language Modelscited by 8
  22. 2024ContextVLM: Zero-Shot and Few-Shot Context Understanding for Autonomous Driving using Vision Language Modelscited by 8
  23. 2025FindAnything: Open-Vocabulary and Object-Centric Mapping for Robot Exploration in Any Environmentcited by 8
  24. 2025ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Modelscited by 8
  25. 2025ViewSRD: 3D Visual Grounding via Structured Multi-View Decompositioncited by 8
  26. 2024Safety Implications of Explainable Artificial Intelligence in End-to-End Autonomous Drivingcited by 7
  27. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  28. 2024INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurancecited by 7
  29. 2025Tracking Meets Large Multimodal Models for Driving Scenario Understandingcited by 7
  30. 2025V3LMA: Visual 3D-enhanced Language Model for Autonomous Drivingcited by 7
  31. 2024Radar Spectra-Language Model for Automotive Scene Parsingcited by 6
  32. 2024Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Drivingcited by 6
  33. 2025Persistent Autoregressive Mapping with Traffic Rules for Autonomous Drivingcited by 6
  34. 2025NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Drivingcited by 6
  35. 2025PADriver: Towards Personalized Autonomous Drivingcited by 5
  36. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 4
  37. 2025CoDriveVLM: VLM-Enhanced Urban Cooperative Dispatching and Motion Planning for Future Autonomous Mobility on Demand Systemscited by 4
  38. 2025ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Modelcited by 4

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations
  2. 2025VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion
  3. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
  4. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
  5. 2025Progressive Bird's Eye View Perception for Safety-Critical Autonomous Driving: A Comprehensive Survey
  6. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
  7. 2026V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
  8. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models