mlsanityData to 7 Oct 2026

ADAPT: Action-aware Driving Caption Transformer

arXiv 2302.00673 ↗First posted 1 Feb 2023cs.CV cs.HCcited by 78

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

Abstract (arXiv)

End-to-end autonomous driving has great potential in the transportation industry. However, the lack of transparency and interpretability of the automatic decision-making process hinders its industrial adoption in practice. There have been some early attempts to use attention maps or cost volume for better model explainability which is difficult for ordinary passengers to understand. To bridge the gap, we propose an end-to-end transformer-based architecture, ADAPT (Action-aware Driving cAPtion Transformer), which provides user-friendly natural language narrations and reasoning for each decision making step of autonomous vehicular control and action. ADAPT jointly trains both the driving caption task and the vehicular control prediction task, through a shared video representation. Experiments on BDD-X (Berkeley DeepDrive eXplanation) dataset demonstrate state-of-the-art performance of the ADAPT framework on both automatic metrics and human evaluation. To illustrate the feasibility of the proposed framework in real-world applications, we build a novel deployable system that takes raw car videos as input and outputs the action narrations and reasoning in real time. The code, models and data are available at https://github.com/jxbbb/ADAPT.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 63 of its 82 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

63 of 82 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  7. 2021Video Swin Transformercited by 1,090
  8. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  9. 2016End to End Learning for Self-Driving Carscited by 1,000
  10. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  11. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  12. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
38 more
  1. 2019Learning by Cheatingcited by 361
  2. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  3. 2018A Short Note about Kinetics-600cited by 331
  4. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  5. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  6. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  7. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  8. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  9. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  10. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  11. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  12. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  13. 2015Sequence to Sequence -- Video to Textcited by 157
  14. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  15. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  16. 2020Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?cited by 139
  17. 2017Detect to Track and Track to Detectcited by 131
  18. 2020A Survey of End-to-End Driving: Architectures and Training Methodscited by 123
  19. 2021Translating Images into Mapscited by 115
  20. 2021Learning to drive from a world on railscited by 114
  21. 2019Multimodal End-to-End Autonomous Drivingcited by 110
  22. 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
  23. 2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 106
  24. 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 106
  25. 2021LookOut: Diverse Multi-Future Prediction and Planning for Self-Drivingcited by 96
  26. 2014Translating Videos to Natural Language Using Deep Recurrent Neural Networkscited by 95
  27. 2018Driving Policy Transfer via Modularity and Abstractioncited by 91
  28. 2022MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queriescited by 90
  29. 2015Describing Videos by Exploiting Temporal Structurecited by 88
  30. 2018Conditional Affordance Learning for Driving in Urban Environmentscited by 70
  31. 2020DSDNet: Deep Structured self-Driving Networkcited by 61
  32. 2015Video Paragraph Captioning Using Hierarchical Recurrent Neural Networkscited by 54
  33. 2015Jointly Modeling Embedding and Translation to Bridge Video and Languagecited by 53
  34. 2016Video Captioning with Transferred Semantic Attributescited by 46
  35. 2020Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understandingcited by 42
  36. 2020Label Efficient Visual Abstractions for Autonomous Drivingcited by 33
  37. 2017Physics Inspired Optimization on Semantic Transfer Features: An Alternative Method for Room Layout Estimationcited by 29
  38. 2021PQ-Transformer: Jointly Parsing 3D Objects and Layouts from Point Cloudscited by 26

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 33; 2025: 22; 2026: 9.

78 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  3. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  4. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  5. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  6. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  7. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  8. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  9. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  10. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  11. 2025GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Drivingcited by 117
  12. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
38 more
  1. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  2. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  3. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  4. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  5. 2024Embodied Understanding of Driving Scenarioscited by 63
  6. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  7. 2024P-MapNet: Far-seeing Map Generator Enhanced by both SDMap and HDMap Priorscited by 54
  8. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  9. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  10. 2024MonoOcc: Digging into Monocular Semantic Occupancy Predictioncited by 43
  11. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  12. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  13. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  14. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  15. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  16. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  17. 2024ChatSUMO: Large Language Model for Automating Traffic Scenario Generation in Simulation of Urban MObilitycited by 34
  18. 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33
  19. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  20. 2025DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Modelcited by 23
  21. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22
  22. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  23. 2025SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Modelscited by 22
  24. 2025AVD2: Accident Video Diffusion for Accident Video Descriptioncited by 20
  25. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Drivingcited by 19
  26. 2023Delving into Shape-aware Zero-shot Semantic Segmentationcited by 16
  27. 2025Embodied Scene Understanding for Vision Language Models via MetaVQAcited by 16
  28. 2025Chameleon: Fast-slow Neuro-symbolic Lane Topology Extractioncited by 16
  29. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  30. 2024Large Language Models for Human-like Autonomous Driving: A Surveycited by 14
  31. 2024Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Modelcited by 14
  32. 2025UncAD: Towards Safe End-to-end Autonomous Driving via Online Map Uncertaintycited by 14
  33. 2024Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMscited by 13
  34. 2024Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputscited by 12
  35. 2026Unifying Language-Action Understanding and Generation for Autonomous Drivingcited by 9
  36. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 8
  37. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  38. 2024Tell Me Where You Are: Multimodal LLMs Meet Place Recognitioncited by 7

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Comparative study of adapting pre-trained models for driving behavior video captioning
  2. 2025DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models
  3. 2025A Survey on Vision-Language-Action Models for Autonomous Driving
  4. 2026Less Language, More Latents: Annotation-Efficient VLAs for Driving
  5. 2026AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving
  6. 2026Vega: Learning to Drive with Natural Language Instructions
  7. 2026SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Driving
  8. 2025DriveBLIP2: Attention-Guided Explanation Generation for Complex Driving Scenarios