mlsanityData to 7 Oct 2026

ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Model

arXiv 2503.13938 ↗First posted 18 Mar 2025cs.CV cs.AIcited by 4

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Comprehensive traffic scene understanding is a foundational capability for Intelligent Transportation Systems (ITS) underpinning applications such as traffic simulation. While VisionLanguage Models (VLMs) have demonstrated strong reasoning potential, their application to Bird's-Eye View (BEV) maps in traffic contexts remains limited by narrow task definitions and scarce annotated data. We introduce ChatBEV-QA, a large-scale BEV VQA benchmark of 137K+ QA pairs, designed to evaluate global scene understanding, vehicle-lane interactions, and vehiclevehicle interactions within complex traffic environments. Building on this, we fine-tune ChatBEV, a specialized VLM that accurately interprets diverse scene understanding queries from BEV maps. To demonstrate downstream utility in ITS applications, we integrate ChatBEV into a language-guided traffic simulation framework. Its global understanding and navigation reasoning provide crucial context-aware guidance, reducing trajectory displacement error by up to 20.9% and scenario collision rates by up to 37.9% over text-only baselines.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 31 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

26 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving3 Oct

Other references · 25

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2022A ConvNet for the 2020scited by 3,545
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  9. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  10. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 239
  11. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  12. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
13 more
  1. 2017Spatial As Deep: Spatial CNN for Traffic Scene Understandingcited by 187
  2. 2020Predicting Semantic Map Representations from Images using Pyramid Occupancy Networkscited by 175
  3. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  4. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 110
  5. 2023Language Prompt for Autonomous Drivingcited by 110
  6. 2023Referring Multi-Object Trackingcited by 91
  7. 2023Language Conditioned Traffic Generationcited by 86
  8. 2021SceneGen: Learning to Generate Realistic Traffic Scenescited by 83
  9. 2020End-to-end Contextual Perception and Prediction with Interaction Transformercited by 54
  10. 2018Object Referring in Videos with Language and Human Gazecited by 40
  11. 2020Deep Multimodal Neural Architecture Searchcited by 34
  12. 2020Understanding Bird's-Eye View of Road Semantics using an Onboard Cameracited by 29
  13. 2024Language-Driven Interactive Traffic Trajectory Generationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  2. 2025Spatial-aware Vision Language Model for Autonomous Drivingcited by 4
  3. 2026More than the Sum: Panorama-Language Models for Adverse Omni-Scenescited by 4
  4. 2026Make Geometry Matter for Spatial Reasoningcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
  2. 2025Embodied Scene Understanding for Vision Language Models via MetaVQA
  3. 2025Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception
  4. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models
  5. 2025V3LMA: Visual 3D-enhanced Language Model for Autonomous Driving
  6. 2026BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations
  7. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving
  8. 2026GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Driving