mlsanityData to 7 Oct 2026

Talk2DM: Enabling Natural Language Querying and Commonsense Reasoning for Vehicle-Road-Cloud Integrated Dynamic Maps with Large Language Models

arXiv 2602.11860 ↗First posted 12 Feb 2026cs.AIcited by 0

On the map: LLM reasoning → Chain-of-thought reasoning for driving

Abstract (arXiv)

Dynamic maps (DM) serve as the fundamental information infrastructure for vehicle-road-cloud (VRC) cooperative autonomous driving in China and Japan. By providing comprehensive traffic scene representations, DM overcome the limitations of standalone autonomous driving systems (ADS), such as physical occlusions. Although DM-enhanced ADS have been successfully deployed in real-world applications in Japan, existing DM systems still lack a natural-language-supported (NLS) human interface, which could substantially enhance human-DM interaction. To address this gap, this paper introduces VRCsim, a VRC cooperative perception (CP) simulation framework designed to generate streaming VRC-CP data. Based on VRCsim, we construct a question-answering data set, VRC-QA, focused on spatial querying and reasoning in mixed-traffic scenes. Building upon VRCsim and VRC-QA, we further propose Talk2DM, a plug-and-play module that extends VRC-DM systems with NLS querying and commonsense reasoning capabilities. Talk2DM is built upon a novel chain-of-prompt (CoP) mechanism that progressively integrates human-defined rules with the commonsense knowledge of large language models (LLMs). Experiments on VRC-QA show that Talk2DM can seamlessly switch across different LLMs while maintaining high NLS query accuracy, demonstrating strong generalization capability. Although larger models tend to achieve higher accuracy, they incur significant efficiency degradation. Our results reveal that Talk2DM, powered by Qwen3:8B, Gemma3:27B, and GPT-oss models, achieves over 93% NLS query accuracy with an average response time of only 2-5 seconds, indicating strong practical potential.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 23 of its 43 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

23 of 43 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  4. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  5. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  6. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  7. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  8. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  9. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  10. 2022CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformerscited by 227
  11. 2022Robot Operating System 2: Design, Architecture, and Uses In The Wildcited by 190
  12. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
11 more
  1. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  2. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  3. 2023Language Prompt for Autonomous Drivingcited by 110
  4. 2023Referring Multi-Object Trackingcited by 91
  5. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  6. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  7. 2018Object Referring in Videos with Language and Human Gazecited by 40
  8. 2024LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Modelscited by 35
  9. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  10. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Promptscited by 18
  11. 2023Vehicle-to-Everything Cooperative Perception for Autonomous Drivingcited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
  2. 2026CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning
  3. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving
  4. 2026Roadside-Cooperative Autonomous Driving: From Data Platform to Vision-Language End-to-End Reasoning
  5. 2026D2-V2X: Depth-Driven Cooperative V2X Reasoning for Autonomous Driving
  6. 2025ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Model
  7. 2025RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System
  8. 2026GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Driving