mlsanityData to 14 Oct 2026

doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation

arXiv 2412.05893 ↗First posted 8 Dec 2024cs.CV cs.AIcited by 15

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

Abstract (arXiv)

Human-interactive robotic systems, particularly autonomous vehicles (AVs), must effectively integrate human instructions into their motion planning. This paper introduces doScenes, a novel dataset designed to facilitate research on human-vehicle instruction interactions, focusing on short-term directives that directly influence vehicle motion. By annotating multimodal sensor data with natural language instructions and referentiality tags, doScenes bridges the gap between instruction and driving response, enabling context-aware and adaptive planning. Unlike existing datasets that focus on ranking or scene-level reasoning, doScenes emphasizes actionable directives tied to static and dynamic scene objects. This framework addresses limitations in prior research, such as reliance on simulated data or predefined action sets, by supporting nuanced and flexible responses in real-world scenarios. This work lays the foundation for developing learning strategies that seamlessly integrate human instructions into autonomous systems, advancing safe and effective human-vehicle collaboration for vision-language navigation. We make our data publicly available at https://www.github.com/rossgreer/doScenes

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 39 of its 48 references that are in the map. First posted on arXiv 8 Dec 2024.

Builds on

39 of 48 references.

References · 39

  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  3. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  4. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  5. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  6. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  7. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  8. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  9. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  10. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  11. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  12. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
27 more
  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  2. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  3. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  4. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  5. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  6. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  7. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 124
  8. 2021Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversalscited by 113
  9. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
  10. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  11. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 68
  12. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  13. 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 48
  14. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotationscited by 31
  15. 2024InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planningcited by 31
  16. 2020Trajectory Prediction in Autonomous Driving with a Lane Heading Auxiliary Losscited by 30
  17. 2024Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Setscited by 25
  18. 2023HandMeThat: Human-Robot Communication in Physical and Social Environmentscited by 20
  19. 2021Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling with Real-World Datacited by 13
  20. 2023Safe Control Transitions: Machine Vision Based Observable Readiness Index and Data-Driven Takeover Time Predictioncited by 12
  21. 2021Predicting Take-over Time for Autonomous Driving with Real-World Data: Robust Data Augmentation, Models, and Evaluationcited by 10
  22. 2018Safe Navigation with Human Instructions in Complex Scenescited by 9
  23. 2023Pedestrian Behavior Maps for Safety Advisories: CHAMP Framework and Real-World Data Analysiscited by 9
  24. 2024Language-Augmented Symbolic Planner for Open-World Task Planningcited by 8
  25. 2023Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognitioncited by 4
  26. 2024NatSGD: A Dataset with Speech, Gestures, and Demonstrations for Robot Learning in Natural Human-Robot Interactioncited by 4
  27. 2024iHERO: Interactive Human-oriented Exploration and Supervision Under Scarce Communicationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehiclescited by 9
  2. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  3. 2026Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Modelscited by 5
  4. 2025Automated Data Curation Using GPS & NLP to Generate Instruction-Action Pairs for Autonomous Vehicle Vision-Language Navigation Datasetscited by 2
  5. 2026Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planningcited by 1
  6. 2026When Stopping Fails: Rethinking Minimal Risk Conditions through Human-Interactive Autonomous Driving for Safe Transportation Systemscited by 1
  7. 2025Less is More: Lean yet Powerful Vision-Language Model for Autonomous Drivingcited by 0
  8. 2025Sentiment Matters: An Analysis of 200 Human-SAV Interactionscited by 0
  9. 2026Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Makingcited by 0
  10. 2026WorldVLM: Combining World Model Forecasting and Vision-Language Reasoningcited by 0
  11. 2026NudgeVAD: Language-Nudged End-to-End Driving via FiLM Residualscited by 0
  12. 2026A DVDrive Approach for doScenes Instructed Driving Challengecited by 0
3 more
  1. 2026A Large-Language-Model Supported Personalized Driving Framework for Lane Change in Highway Scenarioscited by 0
  2. 2026doPlan: A Variable-Horizon Dataset for Multi-Stage Language-Conditioned Planning in Autonomous Drivingcited by 0
  3. 2026Lamarck's Driving School: Discovering Autonomous Driving Training Strategies through Evolutionary Competitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Vega: Learning to Drive with Natural Language Instructions
  2. 2025DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models
  3. 2025CARScenes: Semantic VLM Dataset for Safe Autonomous Driving
  4. 2025NavigScene: Bridging Local Perception and Global Navigation for Beyond-Visual-Range Autonomous Driving
  5. 2026ScenePilot-4K: A Large-Scale First-Person Dataset and Benchmark for Vision-Language Models in Autonomous Driving
  6. 2025From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models
  7. 2025SAFe-Copilot: Unified Shared Autonomy Framework ↗
  8. 2025A Survey on Vision-Language-Action Models for Autonomous Driving