mlsanityData to 7 Oct 2026

Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving

arXiv 2409.06702 ↗First posted 10 Sep 2024cs.CV cs.AIcited by 33

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

Abstract (arXiv)

End-to-end architectures in autonomous driving (AD) face a significant challenge in interpretability, impeding human-AI trust. Human-friendly natural language has been explored for tasks such as driving explanation and 3D captioning. However, previous works primarily focused on the paradigm of declarative interpretability, where the natural language interpretations are not grounded in the intermediate outputs of AD systems, making the interpretations only declarative. In contrast, aligned interpretability establishes a connection between language and the intermediate outputs of AD systems. Here we introduce Hint-AD, an integrated AD-language system that generates language aligned with the holistic perception-prediction-planning outputs of the AD model. By incorporating the intermediate outputs and a holistic token mixer sub-network for effective feature adaptation, Hint-AD achieves desirable accuracy, achieving state-of-the-art results in driving language tasks including driving explanation, 3D dense captioning, and command prediction. To facilitate further study on driving explanation task on nuScenes, we also introduce a human-labeled dataset, Nu-X. Codes, dataset, and models will be publicly available.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 39 of its 51 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

39 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ADAPT: Action-aware Driving Caption Transformer1 Feb

Other references · 38

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  8. 2021Video Swin Transformercited by 1,090
  9. 2022Planning-oriented Autonomous Drivingcited by 1,068
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  12. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
26 more
  1. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  2. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  3. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  4. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  5. 2019Learning by Cheatingcited by 361
  6. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  7. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  8. 2018Learning to Drive in a Daycited by 266
  9. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  10. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  11. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  12. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  13. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  14. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  15. 2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 106
  16. 2021LookOut: Diverse Multi-Future Prediction and Planning for Self-Drivingcited by 96
  17. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  18. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  19. 2024P-MapNet: Far-seeing Map Generator Enhanced by both SDMap and HDMap Priorscited by 54
  20. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  21. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 38
  22. 2023STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimationcited by 36
  23. 2021Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillationcited by 25
  24. 2023From Semi-supervised to Omni-supervised Room Layout Estimation Using Point Cloudscited by 18
  25. 2023DQS3D: Densely-matched Quantization-aware Semi-supervised 3D Detectioncited by 16
  26. 2024Training-Free Model Merging for Multi-target Domain Adaptationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 15.

33 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  3. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  4. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  5. 2025TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Modelscited by 30
  6. 2025AVD2: Accident Video Diffusion for Accident Video Descriptioncited by 20
  7. 2025Challenger: Affordable Adversarial Driving Video Generationcited by 16
  8. 2025InsightDrive: Insight Scene Representation for End-to-End Autonomous Drivingcited by 14
  9. 2025RoboChemist: Long-Horizon and Safety-Compliant Robotic Chemical Experimentationcited by 9
  10. 2026Unifying Language-Action Understanding and Generation for Autonomous Drivingcited by 9
  11. 2024Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Drivingcited by 6
  12. 2025Delving into Mapping Uncertainty for Mapless Trajectory Predictioncited by 6
21 more
  1. 2026Vega: Learning to Drive with Natural Language Instructionscited by 6
  2. 2025Risk Map As Middleware: Towards Interpretable Cooperative End-to-end Autonomous Driving for Risk-Aware Planningcited by 5
  3. 2026LottieGPT: Tokenizing Vector Animation for Autoregressive Generationcited by 5
  4. 2025PosePilot: Steering Camera Pose for Generative World Models with Self-supervised Depthcited by 4
  5. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Processcited by 4
  6. 2025LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Drivingcited by 4
  7. 2026FROST-Drive: Scalable and Efficient End-to-End Driving with a Frozen Vision Encodercited by 4
  8. 2026Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planningcited by 4
  9. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  10. 2026UniUncer: Unified Dynamic Static Uncertainty for End to End Drivingcited by 2
  11. 2026Planning-aligned Token Compression for Long-Context Autonomous Drivingcited by 2
  12. 2024AD-DINO: Attention-Dynamic DINO for Distance-Aware Embodied Reference Understandingcited by 1
  13. 2025HMAD: Advancing E2E Driving with Anchored Offset Proposals and Simulation-Supervised Multi-target Scoringcited by 1
  14. 2026Efficient and Explainable End-to-End Autonomous Driving via Masked Vision-Language-Action Diffusioncited by 1
  15. 2026Grounding Driving VLA via Inverse Kinematicscited by 1
  16. 2026PriorEye: Geospatial Visual Priors for End-to-End Autonomous Drivingcited by 1
  17. 2026Cross-Stage Coherence in Hierarchical Driving VQA: Explicit Baselines and Learned Gated Context Projectorscited by 0
  18. 2026FleetAgent: Teleoperation Assistant for Autonomous Fleets via Vectorized V2N Messagescited by 0
  19. 2026Plug-and-Play Traffic Element Awareness for End-to-End Autonomous Drivingcited by 0
  20. 2026Learning to Explain While Planning: Rule-Aligned Diffusion Planning for Autonomous Drivingcited by 0
  21. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models
  2. 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Driving
  3. 2024Large Language Models for Human-like Autonomous Driving: A Survey
  4. 2025AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving
  5. 2025A Survey on Vision-Language-Action Models for Autonomous Driving
  6. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agents
  7. 2026Driving the Wrong Way: Leveraging Interpretability in End2End Autonomous Driving Models
  8. 2026Rethinking Language's Role in Efficient VLA for Autonomous Vehicles: Toward Smarter, Trustworthy Driving