mlsanityData to 7 Oct 2026

Language Prompt for Autonomous Driving

arXiv 2309.04379 ↗First posted 8 Sep 2023cs.CVcited by 110

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

A new trend in the computer vision community is to capture objects of interest following flexible human command represented by a natural language prompt. However, the progress of using language prompts in driving scenarios is stuck in a bottleneck due to the scarcity of paired prompt-instance data. To address this challenge, we propose the first object-centric language prompt set for driving scenes within 3D, multi-view, and multi-frame space, named NuPrompt. It expands nuScenes dataset by constructing a total of 40,147 language descriptions, each referring to an average of 7.4 object tracklets. Based on the object-text pairs from the new benchmark, we formulate a novel prompt-based driving task, ie, employing a language prompt to predict the described object trajectory across views and frames. Furthermore, we provide a simple end-to-end baseline model based on Transformer, named PromptTrack. Experiments show that our PromptTrack achieves impressive performance on NuPrompt. We hope this work can provide some new insights for the self-driving community. The data and code have been released at https://github.com/wudongming97/Prompt4Driving.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 38 of its 51 references that are in the map. First posted on arXiv 8 Sep 2023.

Builds on

38 of 51 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023Segment Anythingcited by 8,158
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
26 more
  1. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  2. 2016Modeling Context in Referring Expressionscited by 1,054
  3. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  5. 2017Focal Loss for Dense Object Detectioncited by 749
  6. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  7. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  8. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  9. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  10. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  11. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  12. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  13. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  14. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  15. 2019An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detectioncited by 163
  16. 2018Video Object Segmentation with Language Referring Expressionscited by 161
  17. 2020TAO: A Large-Scale Benchmark for Tracking Any Objectcited by 146
  18. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  19. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  20. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  21. 2023Referring Multi-Object Trackingcited by 91
  22. 2019Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Videocited by 60
  23. 2023Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Trackingcited by 54
  24. 2018Object Referring in Videos with Language and Human Gazecited by 40
  25. 2023Type-to-Track: Retrieve Any Object via Prompt-based Trackingcited by 24
  26. 2023The 1st-place Solution for CVPR 2023 OpenLane Topology in Autonomous Driving Challengecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 33; 2025: 48; 2026: 15.

110 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  5. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  6. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  7. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  8. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  9. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  10. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  11. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  12. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
38 more
  1. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  2. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  3. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
  4. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  5. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  6. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  7. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  8. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  9. 2024Embodied Understanding of Driving Scenarioscited by 63
  10. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  11. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  12. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  13. 2024LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Modelscited by 35
  14. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  15. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  16. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  17. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  18. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  19. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  20. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  21. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 26
  22. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  23. 2025Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learningcited by 25
  24. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  25. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 23
  26. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22
  27. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  28. 2024Bootstrapping Referring Multi-Object Trackingcited by 21
  29. 2024EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Drivingcited by 18
  30. 2024Data-Centric Evolution in Autonomous Driving: A Comprehensive Survey of Big Data System, Data Mining, and Closed-Loop Technologiescited by 17
  31. 2024Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehensioncited by 17
  32. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  33. 2025Embodied Scene Understanding for Vision Language Models via MetaVQAcited by 16
  34. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  35. 2024Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Modelcited by 14
  36. 2024Large Language Models Powered Context-aware Motion Prediction in Autonomous Drivingcited by 13
  37. 2025RefAV: Towards Planning-Centric Scenario Miningcited by 13
  38. 2025NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Drivingcited by 12

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Vega: Learning to Drive with Natural Language Instructions
  2. 2024LeGo-Drive: Language-enhanced Goal-oriented Closed-Loop End-to-End Autonomous Driving
  3. 2026Less Language, More Latents: Annotation-Efficient VLAs for Driving
  4. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation
  5. 2025Enhancing Vision-Language Models for Autonomous Driving through Task-Specific Prompting and Spatial Reasoning
  6. 2026BLUE: Toward Better Language Use in Efficient Vision-Language-Action Models for Autonomous Driving
  7. 2024EMMA: End-to-End Multimodal Model for Autonomous Driving
  8. 2025Less is More: Lean yet Powerful Vision-Language Model for Autonomous Driving