Efficient and Explainable End-to-End Autonomous Driving via Masked Vision-Language-Action Diffusion
On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection
also in 2 more
Abstract (arXiv)
Large Language Models (LLMs) and Vision-Language Models (VLMs) have emerged as promising candidates for end-to-end autonomous driving. However, these models typically face challenges in inference latency, action precision, and explainability. Existing autoregressive approaches struggle with slow token-by-token generation, while prior diffusion-based planners often rely on verbose, general-purpose language tokens that lack explicit geometric structure. In this work, we propose Masked Vision-Language-Action Diffusion for Autonomous Driving (MVLAD-AD), a novel framework designed to bridge the gap between efficient planning and semantic explainability via a masked vision-language-action diffusion model. Unlike methods that force actions into the language space, we introduce a discrete action tokenization strategy that constructs a compact codebook of kinematically feasible waypoints from real-world driving distributions. Moreover, we propose geometry-aware embedding learning to ensure that embeddings in the latent space approximate physical geometric metrics. Finally, an action-priority decoding strategy is introduced to prioritize trajectory generation. Extensive experiments on nuScenes and derived benchmarks demonstrate that MVLAD-AD achieves superior efficiency and outperforms state-of-the-art autoregressive and diffusion baselines in planning precision, while providing high-fidelity and explainable reasoning.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 42 references that are in the map. First posted on arXiv 24 Feb 2026.
Builds on
38 of 42 references.
Closest ancestors · 6
Same direction, newest first
- 2026HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Driving1 Feb
- 2025ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving18 Aug
- 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving19 Dec
- 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
- 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
- 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
Other references · 32
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
- 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
- 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
- 2022Planning-oriented Autonomous Drivingcited by 1,068
- 2019Attention is not Explanationcited by 916
20 more
- 2025Large Language Diffusion Modelscited by 872
- 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
- 2023GPT-Driver: Learning to Drive with GPTcited by 344
- 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
- 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
- 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
- 2018Textual Explanations for Self-Driving Vehiclescited by 226
- 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
- 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
- 2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 106
- 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
- 2021Explanations in Autonomous Driving: A Surveycited by 80
- 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
- 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 46
- 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
- 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33
- 2024DriveGPT: Scaling Autoregressive Behavior Models for Drivingcited by 29
- 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
- 2023Textual Explanations for Automated Commentary Drivingcited by 6
- 2025ALN-P3: Unified Language Alignment for Perception, Prediction, and Planning in Autonomous Drivingcited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers
- 2026VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving
- 2026Continuous Actions from Discrete Minds: Latent-Aligned Planning for End-to-End Autonomous Driving
- 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
- 2025Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving
- 2026S-squared-VLA: Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving
- 2025WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Driving
- 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model