mlsanityData to 14 Oct 2026

Grounding Human-to-Vehicle Advice for Self-driving Vehicles

arXiv 1911.06978 ↗First posted 16 Nov 2019cs.CVcited by 71

Not on the map. Why?

Cited by 35 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent success suggests that deep neural control networks are likely to be a key component of self-driving vehicles. These networks are trained on large datasets to imitate human actions, but they lack semantic understanding of image contents. This makes them brittle and potentially unsafe in situations that do not match training data. Here, we propose to address this issue by augmenting training data with natural language advice from a human. Advice includes guidance about what to do and where to attend. We present the first step toward advice giving, where we train an end-to-end vehicle controller that accepts advice. The controller adapts the way it attends to the scene (visual attention) and the control (steering and speed). Attention mechanisms tie controller behavior to salient objects in the advice. We evaluate our model on a novel advisable driving dataset with manually annotated human-to-vehicle advice called Honda Research Institute-Advice Dataset (HAD). We show that taking advice improves the performance of the end-to-end network, while the network cues on a variety of visual features that are provided by advice. The dataset is available at https://usa.honda-ri.com/HAD.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 6; 2023: 17; 2024: 17; 2025: 15; 2026: 8.

71 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  2. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  4. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  5. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 143
  6. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  7. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  8. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  9. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 103
  10. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 93
  11. 2021OadTR: Online Action Detection with Transformerscited by 83
  12. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 78
38 more
  1. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 77
  2. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 66
  3. 2024Embodied Understanding of Driving Scenarioscited by 63
  4. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  5. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 50
  6. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  7. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  8. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 33
  9. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  10. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  11. 2021Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterizationcited by 22
  12. 2024Probing Multimodal LLMs as World Models for Drivingcited by 20
  13. 2021Salient Objects in Cluttercited by 19
  14. 2025Embodied Scene Understanding for Vision Language Models via MetaVQAcited by 19
  15. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  16. 2025A Survey on Efficient Vision-Language Modelscited by 19
  17. 2021PSI: A Benchmark for Human Interpretation and Response in Traffic Interactionscited by 17
  18. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  19. 2023Explaining Autonomous Driving Actions with Visual Question Answeringcited by 16
  20. 2024On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validationcited by 15
  21. 2025Context-Enhanced Memory-Refined Transformer for Online Action Detectioncited by 13
  22. 2022SOTVerse: A User-defined Task Space of Single Object Trackingcited by 10
  23. 2022Correspondence Matters for Video Referring Expression Comprehensioncited by 10
  24. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  25. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 10
  26. 2021Behavioral Research and Practical Models of Drivers' Attentioncited by 9
  27. 2023Driving through the Concept Gridlock: Unraveling Explainability Bottlenecks in Automated Drivingcited by 9
  28. 2025CoopReflect: Towards Natural Language Communication for Cooperative Autonomous Driving via Multi-Agent Learningcited by 7
  29. 2025CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refinecited by 7
  30. 2021Learning to Discriminate Information for Online Action Detection: Analysis and Applicationcited by 6
  31. 2023Textual Explanations for Automated Commentary Drivingcited by 6
  32. 2023SAFE: Saliency-Aware Counterfactual Explanations for DNN-based Automated Driving Systemscited by 5
  33. 2025H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Drivingcited by 5
  34. 2022Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attentioncited by 4
  35. 2024ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videoscited by 4
  36. 2024Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Contextcited by 4
  37. 2026See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selectioncited by 4
  38. 2023Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillationcited by 3

50 most cited of 71 All 71 (JSON)