Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
On the map: Vision-language-action models → VLA for autonomous driving
also in 1 more
Abstract (arXiv)
Autonomous driving has long relied on modular "Perception-Decision-Action" pipelines, where hand-crafted interfaces and rule-based components often break down in complex or long-tailed scenarios. Their cascaded design further propagates perception errors, degrading downstream planning and control. Vision-Action (VA) models address some limitations by learning direct mappings from visual inputs to actions, but they remain opaque, sensitive to distribution shifts, and lack structured reasoning or instruction-following capabilities. Recent progress in Large Language Models (LLMs) and multimodal learning has motivated the emergence of Vision-Language-Action (VLA) frameworks, which integrate perception with language-grounded decision making. By unifying visual understanding, linguistic reasoning, and actionable outputs, VLAs offer a pathway toward more interpretable, generalizable, and human-aligned driving policies. This work provides a structured characterization of the emerging VLA landscape for autonomous driving. We trace the evolution from early VA approaches to modern VLA frameworks and organize existing methods into two principal paradigms: End-to-End VLA, which integrates perception, reasoning, and planning within a single model, and Dual-System VLA, which separates slow deliberation (via VLMs) from fast, safety-critical execution (via planners). Within these paradigms, we further distinguish subclasses such as textual vs. numerical action generators and explicit vs. implicit guidance mechanisms. We also summarize representative datasets and benchmarks for evaluating VLA-based driving systems and highlight key challenges and open directions, including robustness, interpretability, and instruction fidelity. Overall, this work aims to establish a coherent foundation for advancing human-compatible autonomous driving systems.
Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in VLA for autonomous driving, one of 13 directions in Vision-language-action models. Builds on 301 of its 337 references that are in the map. First posted on arXiv 18 Dec 2025.
Builds on
301 of 337 references.
Closest ancestors · 14
Same direction, newest first
- 2025Latent Chain-of-Thought World Modeling for End-to-End Driving11 Dec
- 2025Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving25 Nov
- 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail30 Oct
- 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving14 Oct
- 2025Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving24 Sep
- 2025OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision6 Sep
- 2025FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning31 Jul
- 2025A Survey on Vision-Language-Action Models for Autonomous Driving30 Jun
- 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning16 Jun
- 2025DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models6 Jun
- 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models29 May
- 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving23 May
2 more
Other references · 287
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2023Visual Instruction Tuningcited by 7,869
38 more
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2017Neural Discrete Representation Learningcited by 4,712
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2022A ConvNet for the 2020scited by 3,579
- 2023Qwen Technical Reportcited by 3,344
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
- 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
- 2018World Modelscited by 1,513
- 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
- 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
- 2022Planning-oriented Autonomous Drivingcited by 1,087
- 2016End to End Learning for Self-Driving Carscited by 1,009
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
- 2022Autoregressive Image Generation using Residual Quantizationcited by 722
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
- 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
- 2024PaliGemma: A versatile 3B VLM for transfercited by 696
- 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
- 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
- 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
50 most cited of 287 All 287 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 32.
33 papers.
Sort:
By area:
- 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 20
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 14
- 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
- 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 13
- 2026Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Drivingcited by 11
- 2026Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanationcited by 10
- 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Drivingcited by 9
- 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compressioncited by 6
- 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
- 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
- 2026From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generationcited by 4
- 2026MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphscited by 3
21 more
- 2026DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Drivingcited by 2
- 2026DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Drivingcited by 2
- 2026Towards Safe Mobility: A Unified Transportation Foundation Model enabled by Open-Ended Vision-Language Datasetcited by 2
- 2026SafeAlign-VLA: A Negative-Enhanced Safe Alignment Framework for Risk-Aware Autonomous Drivingcited by 2
- 2026EventDrive: Event Cameras for Vision-Language Driving Intelligencecited by 2
- 2026Post-Training in End-to-End Autonomous Drivingcited by 2
- 2026From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Drivingcited by 1
- 2026CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Drivingcited by 1
- 2026Grounding Driving VLA via Inverse Kinematicscited by 1
- 2026DriveVer: Lightweight Trajectory Evaluator as Test-Time Verifier for Autonomous Drivingcited by 1
- 2026HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Drivingcited by 1
- 2026CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Modelscited by 1
- 2026Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Makingcited by 0
- 2026From Steering to Pedalling: Do Autonomous Driving VLMs Generalize to Cyclist-Assistive Spatial Perception and Planning?cited by 0
- 2026Safety Case Patterns for VLA-based driving systems: Insights from SimLingocited by 0
- 2026LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Modelcited by 0
- 2026Driving on Memorycited by 0
- 2026Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Drivingcited by 0
- 2026CrashDiffuser: VLM-Guided Collision Intent Reasoning for Fine-Grained Safety-Critical Traffic Scenario Generationcited by 0
- 2026Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Drivingcited by 0
- 2026VITAL: From Visual Motion to Precise Actioncited by 0
- 2026VITAL: From Visual Motion to Precise Action14 Oct 2026
- 2026Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving7 Oct 2026
- 2026CrashDiffuser: VLM-Guided Collision Intent Reasoning for Fine-Grained Safety-Critical Traffic Scenario Generation2 Sep 2026
- 2026Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving31 Aug 2026
- 2026Driving on Memory31 Aug 2026
- 2026CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models7 Aug 2026
- 2026HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Driving23 Jul 2026
- 2026Post-Training in End-to-End Autonomous Driving9 Jul 2026
- 2026DriveVer: Lightweight Trajectory Evaluator as Test-Time Verifier for Autonomous Driving1 Jul 2026
- 2026EventDrive: Event Cameras for Vision-Language Driving Intelligence16 Jun 2026
- 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving29 May 2026
- 2026LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model21 May 2026
21 more
- 2026Grounding Driving VLA via Inverse Kinematics20 May 2026
- 2026SafeAlign-VLA: A Negative-Enhanced Safe Alignment Framework for Risk-Aware Autonomous Driving19 May 2026
- 2026CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Driving17 May 2026
- 2026From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation9 May 2026
- 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms26 Apr 2026
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond24 Apr 2026
- 2026Towards Safe Mobility: A Unified Transportation Foundation Model enabled by Open-Ended Vision-Language Dataset24 Apr 2026
- 2026Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation20 Apr 2026
- 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation13 Apr 2026
- 2026DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Driving1 Apr 2026
- 2026Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving28 Mar 2026
- 2026DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Driving20 Mar 2026
- 2026Safety Case Patterns for VLA-based driving systems: Insights from SimLingo16 Mar 2026
- 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation16 Mar 2026
- 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving24 Feb 2026
- 2026MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs20 Feb 2026
- 2026From Steering to Pedalling: Do Autonomous Driving VLMs Generalize to Cyclist-Assistive Spatial Perception and Planning?11 Feb 2026
- 2026From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving11 Feb 2026
- 2026Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making7 Feb 2026
- 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression29 Jan 2026
- 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systems30 Dec 2025
- 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Drivingcited by 9
- 2026SafeAlign-VLA: A Negative-Enhanced Safe Alignment Framework for Risk-Aware Autonomous Drivingcited by 2
- 2026EventDrive: Event Cameras for Vision-Language Driving Intelligencecited by 2
- 2026Post-Training in End-to-End Autonomous Drivingcited by 2
- 2026Grounding Driving VLA via Inverse Kinematicscited by 1
- 2026DriveVer: Lightweight Trajectory Evaluator as Test-Time Verifier for Autonomous Drivingcited by 1
- 2026From Steering to Pedalling: Do Autonomous Driving VLMs Generalize to Cyclist-Assistive Spatial Perception and Planning?cited by 0
- 2026Safety Case Patterns for VLA-based driving systems: Insights from SimLingocited by 0
- 2026Driving on Memorycited by 0
- 2026Driving on Memory31 Aug 2026
- 2026Post-Training in End-to-End Autonomous Driving9 Jul 2026
- 2026DriveVer: Lightweight Trajectory Evaluator as Test-Time Verifier for Autonomous Driving1 Jul 2026
- 2026EventDrive: Event Cameras for Vision-Language Driving Intelligence16 Jun 2026
- 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving29 May 2026
- 2026Grounding Driving VLA via Inverse Kinematics20 May 2026
- 2026SafeAlign-VLA: A Negative-Enhanced Safe Alignment Framework for Risk-Aware Autonomous Driving19 May 2026
- 2026Safety Case Patterns for VLA-based driving systems: Insights from SimLingo16 Mar 2026
- 2026From Steering to Pedalling: Do Autonomous Driving VLMs Generalize to Cyclist-Assistive Spatial Perception and Planning?11 Feb 2026
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 14
- 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
- 2026Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Drivingcited by 11
- 2026Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanationcited by 10
- 2026DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Drivingcited by 2
- 2026DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Drivingcited by 2
- 2026HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Drivingcited by 1
- 2026LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Modelcited by 0
- 2026VITAL: From Visual Motion to Precise Actioncited by 0
- 2026VITAL: From Visual Motion to Precise Action14 Oct 2026
- 2026HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Driving23 Jul 2026
- 2026LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model21 May 2026
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond24 Apr 2026
- 2026Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation20 Apr 2026
- 2026DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Driving1 Apr 2026
- 2026Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving28 Mar 2026
- 2026DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Driving20 Mar 2026
- 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation16 Mar 2026
- 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
- 2026MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphscited by 3
- 2026Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Drivingcited by 0
- 2026CrashDiffuser: VLM-Guided Collision Intent Reasoning for Fine-Grained Safety-Critical Traffic Scenario Generationcited by 0
- 2026Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Drivingcited by 0
- 2026Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving7 Oct 2026
- 2026CrashDiffuser: VLM-Guided Collision Intent Reasoning for Fine-Grained Safety-Critical Traffic Scenario Generation2 Sep 2026
- 2026Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving31 Aug 2026
- 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation13 Apr 2026
- 2026MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs20 Feb 2026
- 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 20
- 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compressioncited by 6
- 2026From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Drivingcited by 1
- 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving24 Feb 2026
- 2026From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving11 Feb 2026
- 2026On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression29 Jan 2026
- 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 13
- 2026CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Drivingcited by 1
- 2026CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Modelscited by 1
- 2026CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models7 Aug 2026
- 2026CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Driving17 May 2026
- 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms26 Apr 2026
- 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
- 2026From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generationcited by 4
- 2026Towards Safe Mobility: A Unified Transportation Foundation Model enabled by Open-Ended Vision-Language Datasetcited by 2
- 2026Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Makingcited by 0
- 2026From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation9 May 2026
- 2026Towards Safe Mobility: A Unified Transportation Foundation Model enabled by Open-Ended Vision-Language Dataset24 Apr 2026
- 2026Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making7 Feb 2026
- 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systems30 Dec 2025
Similar, not cited
Close in wording, no citation either way
- 2026Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Driving
- 2026Learning Vision-Language-Action World Models for Autonomous Driving
- 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
- 2026Unifying Language-Action Understanding and Generation for Autonomous Driving
- 2026Teaching Vision-Language-Action Models What to See and Where to Look
- 2026Does Visual Information Play a Decisive Role in Vision-Language-Action Model Driving Behavior?
- 2026RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillation
- 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges