# Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2505.23757 · HTML: https://mlsanity.com/paper/2505.23757/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Placed in VLA for autonomous driving, one of 13 directions in Vision-language-action models.
- Builds on 73 of its 85 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- Vision-language-action models / [VLA for autonomous driving](https://mlsanity.com/vla/vla-for-autonomous-driving/)
- End-to-end autonomous driving / [Reasoning VLA and VLM driving policies](https://mlsanity.com/driving/reasoning-vla-and-vlm-driving-policies/)

## Builds on (most cited first, 30 of 73)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024)
- [NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://mlsanity.com/paper/2305.14836/) (2023)
- [Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving](https://mlsanity.com/paper/2406.03877/) (2024)
- [Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?](https://mlsanity.com/paper/2312.03031/) (2023)
- [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023)
- [Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving](https://mlsanity.com/paper/2310.01957/) (2023)

## Built on by (most cited first, 30 of 60)

- [Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail](https://mlsanity.com/paper/2511.00088/) (2025)
- [A Survey on Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2506.24044/) (2025)
- [AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2509.13769/) (2025)
- [Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments](https://mlsanity.com/paper/2605.30280/) (2026)
- [OmniNWM: Omniscient Driving Navigation World Models](https://mlsanity.com/paper/2510.18313/) (2025)
- [Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future](https://mlsanity.com/paper/2512.16760/) (2025)
- [TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Models](https://mlsanity.com/paper/2509.07962/) (2025)
- [ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving](https://mlsanity.com/paper/2508.11428/) (2025)
- [DriveVA: Video Action Models are Zero-Shot Drivers](https://mlsanity.com/paper/2604.04198/) (2026)
- [FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning](https://mlsanity.com/paper/2507.23318/) (2025)
- [Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failures](https://mlsanity.com/paper/2603.01063/) (2026)
- [UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving](https://mlsanity.com/paper/2604.02190/) (2026)
- [NoRD: A Data-Efficient Vision-Language-Action Model that Drives without Reasoning](https://mlsanity.com/paper/2602.21172/) (2026)
- [Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving](https://mlsanity.com/paper/2511.19912/) (2025)
- [dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning](https://mlsanity.com/paper/2512.04459/) (2025)
- [ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Driving](https://mlsanity.com/paper/2512.22939/) (2025)
- [Learning Vision-Language-Action World Models for Autonomous Driving](https://mlsanity.com/paper/2604.09059/) (2026)
- [CLOVER: Closed-Loop Value Estimation and Ranking for End-to-End Autonomous Driving Planning](https://mlsanity.com/paper/2605.15120/) (2026)
- [UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving](https://mlsanity.com/paper/2512.09864/) (2025)
- [FLARE: Learning Future-Aware Latent Representations from Vision-Language Models for Autonomous Driving](https://mlsanity.com/paper/2601.05611/) (2026)
- [RoboChemist: Long-Horizon and Safety-Compliant Robotic Chemical Experimentation](https://mlsanity.com/paper/2509.08820/) (2025)
- [MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Cases](https://mlsanity.com/paper/2509.20843/) (2025)
- [Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving](https://mlsanity.com/paper/2511.19221/) (2025)
- [nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving](https://mlsanity.com/paper/2605.31572/) (2026)
- [Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving](https://mlsanity.com/paper/2609.00111/) (2026)
- [CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refine](https://mlsanity.com/paper/2509.15968/) (2025)
- [MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving](https://mlsanity.com/paper/2605.12624/) (2026)
- [Drive My Way: Preference Alignment of Vision-Language-Action Model for Personalized Driving](https://mlsanity.com/paper/2603.25740/) (2026)
- [LottieGPT: Tokenizing Vector Animation for Autoregressive Generation](https://mlsanity.com/paper/2604.11792/) (2026)
- [Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2509.10570/) (2025)

## Abstract (arXiv)

Vision-Language-Action (VLA) models for autonomous driving show promise but falter in unstructured corner case scenarios, largely due to a scarcity of targeted benchmarks. To address this, we introduce Impromptu VLA. Our core contribution is the Impromptu VLA Dataset: over 80,000 meticulously curated video clips, distilled from over 2M source clips sourced from 8 open-source large-scale datasets. This dataset is built upon our novel taxonomy of four challenging unstructured categories and features rich, planning-oriented question-answering annotations and action trajectories. Crucially, experiments demonstrate that VLAs trained with our dataset achieve substantial performance gains on established benchmarks--improving closed-loop NeuroNCAP scores and collision rates, and reaching near state-of-the-art L2 accuracy in open-loop nuScenes trajectory prediction. Furthermore, our Q&A suite serves as an effective diagnostic, revealing clear VLM improvements in perception, prediction, and planning. Our code, data and models are available at https://github.com/ahydchh/Impromptu-VLA.