mlsanityData to 14 Oct 2026

Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

arXiv 2505.23757 ↗First posted 29 May 2025cs.CVcited by 60

On the map: Vision-language-action models → VLA for autonomous driving

also in 1 more
Abstract (arXiv)

Vision-Language-Action (VLA) models for autonomous driving show promise but falter in unstructured corner case scenarios, largely due to a scarcity of targeted benchmarks. To address this, we introduce Impromptu VLA. Our core contribution is the Impromptu VLA Dataset: over 80,000 meticulously curated video clips, distilled from over 2M source clips sourced from 8 open-source large-scale datasets. This dataset is built upon our novel taxonomy of four challenging unstructured categories and features rich, planning-oriented question-answering annotations and action trajectories. Crucially, experiments demonstrate that VLAs trained with our dataset achieve substantial performance gains on established benchmarks--improving closed-loop NeuroNCAP scores and collision rates, and reaching near state-of-the-art L2 accuracy in open-loop nuScenes trajectory prediction. Furthermore, our Q&A suite serves as an effective diagnostic, revealing clear VLM improvements in perception, prediction, and planning. Our code, data and models are available at https://github.com/ahydchh/Impromptu-VLA.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in VLA for autonomous driving, one of 13 directions in Vision-language-action models. Builds on 73 of its 85 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

73 of 85 references.

References · 73

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  9. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  10. 2022Planning-oriented Autonomous Drivingcited by 1,087
  11. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  12. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
38 more
  1. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  2. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  4. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  5. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  6. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  7. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  8. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 377
  9. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  10. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 354
  11. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  12. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  13. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 289
  14. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  15. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 252
  16. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  17. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  18. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  19. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  20. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  21. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  22. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 194
  23. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  24. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  25. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 166
  26. 2018IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environmentscited by 163
  27. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  28. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 144
  29. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 142
  30. 2019Detecting the Unexpected via Image Resynthesiscited by 137
  31. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 123
  32. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
  33. 2016Lost and Found: Detecting Small Road Hazards for Self-Driving Vehiclescited by 117
  34. 2023Language Prompt for Autonomous Drivingcited by 111
  35. 2021Pixel-wise Anomaly Detection in Complex Driving Scenescited by 109
  36. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 106
  37. 2020Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentationcited by 96
  38. 2023Referring Multi-Object Trackingcited by 93

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 41.

60 papers.

Sort:

By area:

  1. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 146
  2. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  3. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 58
  4. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 44
  5. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 33
  6. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  7. 2025TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Modelscited by 32
  8. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 25
  9. 2026DriveVA: Video Action Models are Zero-Shot Driverscited by 23
  10. 2025FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruningcited by 20
  11. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 19
  12. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 19
38 more
  1. 2026NoRD: A Data-Efficient Vision-Language-Action Model that Drives without Reasoningcited by 15
  2. 2025Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Drivingcited by 14
  3. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoningcited by 14
  4. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Drivingcited by 11
  5. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  6. 2026CLOVER: Closed-Loop Value Estimation and Ranking for End-to-End Autonomous Driving Planningcited by 11
  7. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  8. 2026FLARE: Learning Future-Aware Latent Representations from Vision-Language Models for Autonomous Drivingcited by 10
  9. 2025RoboChemist: Long-Horizon and Safety-Compliant Robotic Chemical Experimentationcited by 9
  10. 2025MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Casescited by 9
  11. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  12. 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Drivingcited by 9
  13. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 9
  14. 2025CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refinecited by 7
  15. 2026MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Drivingcited by 6
  16. 2026Drive My Way: Preference Alignment of Vision-Language-Action Model for Personalized Drivingcited by 5
  17. 2026LottieGPT: Tokenizing Vector Animation for Autoregressive Generationcited by 5
  18. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Surveycited by 3
  19. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  20. 2026EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillationcited by 3
  21. 2026OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Modelscited by 3
  22. 2026HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generationcited by 3
  23. 2026Planning-aligned Token Compression for Long-Context Autonomous Drivingcited by 3
  24. 2026VLGA: Vision-Language-Geometry-Action Models for Autonomous Drivingcited by 3
  25. 2026UniUncer: Unified Dynamic Static Uncertainty for End to End Drivingcited by 2
  26. 2026DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Drivingcited by 2
  27. 2026EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Drivingcited by 2
  28. 2026AsyncShield: A Plug-and-Play Edge Adapter for Asynchronous Cloud-based VLA Navigationcited by 2
  29. 2026EventDrive: Event Cameras for Vision-Language Driving Intelligencecited by 2
  30. 2025A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approachcited by 1
  31. 2026The Blind Spot of Adaptation: Quantifying and Mitigating Forgetting in Fine-tuned Driving Modelscited by 1
  32. 2026Grounding Driving VLA via Inverse Kinematicscited by 1
  33. 2026Teaching Vision-Language-Action Models What to See and Where to Lookcited by 1
  34. 2026WCog-VLA: A Dual-Level World-Cognitive Vision-Language-Action Model for End-to-End Autonomous Drivingcited by 1
  35. 2026Does Visual Information Play a Decisive Role in Vision-Language-Action Model Driving Behavior?cited by 0
  36. 2026GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Drivingcited by 0
  37. 2026PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Drivingcited by 0
  38. 2026PrismAD: Decoupled Planning via Semantic Mixture-of-Planners for End-to-End Autonomous Drivingcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models
  2. 2026FIVE-VLA: Fast and EffectIVE Autonomous Driving with Recurrent Action Memory
  3. 2026Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Driving
  4. 2026StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Driving
  5. 2026Less Language, More Latents: Annotation-Efficient VLAs for Driving
  6. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving
  7. 2026RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillation
  8. 2026Unifying Language-Action Understanding and Generation for Autonomous Driving