mlsanityData to 7 Oct 2026

Talk2Car: Taking Control of Your Self-Driving Car

arXiv 1909.10838 ↗First posted 24 Sep 2019cs.AI cs.CL cs.ROcited by 141

Not on the map. Why?

Cited by 59 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A long-term goal of artificial intelligence is to have an agent execute commands communicated through natural language. In many cases the commands are grounded in a visual environment shared by the human who gives the command and the agent. Execution of the command then requires mapping the command into the physical visual space, after which the appropriate action can be taken. In this paper we consider the former. Or more specifically, we consider the problem in an autonomous driving setting, where a passenger requests an action that can be associated with an object found in a street scene. Our work presents the Talk2Car dataset, which is the first object referral dataset that contains commands written in natural language for self-driving cars. We provide a detailed comparison with related datasets such as ReferIt, RefCOCO, RefCOCO+, RefCOCOg, Cityscape-Ref and CLEVR-Ref. Additionally, we include a performance analysis using strong state-of-the-art models. The results show that the proposed object referral task is a challenging one for which the models show promising results but still require additional research in natural language processing, computer vision and the intersection of these fields. The dataset can be found on our website: http://macchina-ai.eu/

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 28; 2024: 32; 2025: 43; 2026: 25.

141 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  5. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  6. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  7. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  8. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  9. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  10. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  11. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  12. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
38 more
  1. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  2. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 121
  3. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  4. 2023Language Prompt for Autonomous Drivingcited by 110
  5. 2023Referring Multi-Object Trackingcited by 91
  6. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  7. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  8. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  9. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  10. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  11. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  12. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  13. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  14. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  15. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  16. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  17. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  18. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  19. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  20. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  21. 2023GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Modelscited by 31
  22. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotationscited by 31
  23. 2024Language-Image Models with 3D Understandingcited by 31
  24. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  25. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  26. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Modelcited by 28
  27. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  28. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  29. 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 22
  30. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  31. 2024Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Drivingcited by 22
  32. 2024Bootstrapping Referring Multi-Object Trackingcited by 21
  33. 2022Ground then Navigate: Language-guided Navigation in Dynamic Scenescited by 20
  34. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20
  35. 2023Language-Guided 3D Object Detection in Point Cloud for Autonomous Drivingcited by 19
  36. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Drivingcited by 19
  37. 2025A Survey on Efficient Vision-Language Modelscited by 19
  38. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17

50 most cited of 141 All 141 (JSON)