mlsanityData to 7 Oct 2026

CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving

arXiv 2408.10845 ↗First posted 19 Aug 2024cs.CVcited by 74

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

Abstract (arXiv)

Autonomous driving, particularly navigating complex and unanticipated scenarios, demands sophisticated reasoning and planning capabilities. While Multi-modal Large Language Models (MLLMs) offer a promising avenue for this, their use has been largely confined to understanding complex environmental contexts or generating high-level driving commands, with few studies extending their application to end-to-end path planning. A major research bottleneck is the lack of large-scale annotated datasets encompassing vision, language, and action. To address this issue, we propose CoVLA (Comprehensive Vision-Language-Action) Dataset, an extensive dataset comprising real-world driving videos spanning more than 80 hours. This dataset leverages a novel, scalable approach based on automated data processing and a caption generation pipeline to generate accurate driving trajectories paired with detailed natural language descriptions of driving environments and maneuvers. This approach utilizes raw in-vehicle sensor data, allowing it to surpass existing datasets in scale and annotation richness. Using CoVLA, we investigate the driving capabilities of MLLMs that can handle vision, language, and action in a variety of driving scenarios. Our results illustrate the strong proficiency of our model in generating coherent language and action outputs, emphasizing the potential of Vision-Language-Action (VLA) models in the field of autonomous driving. This dataset establishes a framework for robust, interpretable, and data-driven autonomous driving systems by providing a comprehensive platform for training and evaluating VLA models, contributing to safer and more reliable self-driving vehicles. The dataset is released for academic purpose.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 44 of its 67 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

44 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ADAPT: Action-aware Driving Caption Transformer1 Feb

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  11. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
31 more
  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  3. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  4. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  5. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  6. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  7. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  8. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  9. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  10. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  11. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 239
  12. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  13. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  14. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  15. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  16. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  17. 2017CityPersons: A Diverse Dataset for Pedestrian Detectioncited by 171
  18. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  19. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  20. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  21. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  22. 2018Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoningcited by 102
  23. 2020Explainable Object-induced Action Decision for Autonomous Vehiclescited by 88
  24. 2019Grounding Human-to-Vehicle Advice for Self-driving Vehiclescited by 71
  25. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  26. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  27. 2021On Exposing the Challenging Long Tail in Future Prediction of Traffic Actorscited by 48
  28. 2022Long-Tail Prediction Uncertainty Aware Trajectory Planning for Self-driving Vehiclescited by 22
  29. 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 22
  30. 2023What Truly Matters in Trajectory Prediction for Autonomous Driving?cited by 17
  31. 2024Data-Centric Evolution in Autonomous Driving: A Comprehensive Survey of Big Data System, Data Mining, and Closed-Loop Technologiescited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 37; 2026: 32.

74 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  3. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  4. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  5. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  6. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  7. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  8. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  9. 2025WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarioscited by 60
  10. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  11. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  12. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
38 more
  1. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Drivingcited by 34
  2. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  3. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Modelcited by 28
  4. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  5. 2026Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Drivingcited by 26
  6. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Drivingcited by 20
  7. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  8. 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planningcited by 17
  9. 2024Generative Emergent Communication: Large Language Model is a Collective World Modelcited by 15
  10. 2026SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Modelcited by 15
  11. 2024ACT-Bench: Towards Action Controllable World Models for Autonomous Drivingcited by 14
  12. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 12
  13. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  14. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarioscited by 11
  15. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  16. 2025DriveQA: Passing the Driving Knowledge Testcited by 10
  17. 2025Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehiclescited by 9
  18. 2025TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videoscited by 9
  19. 2025cVLA: Towards Efficient Camera-Space VLAscited by 8
  20. 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Drivingcited by 8
  21. 2025V3LMA: Visual 3D-enhanced Language Model for Autonomous Drivingcited by 7
  22. 2025CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refinecited by 7
  23. 2026Inject Once Survive Later: Backdooring Vision-Language-Action Models to Persist Through Downstream Fine-tuningcited by 7
  24. 2025CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Drivingcited by 5
  25. 2026Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Modelscited by 5
  26. 2026HiST-VLA: A Hierarchical Spatio-Temporal Vision-Language-Action Model for End-to-End Autonomous Drivingcited by 5
  27. 2025Bring the Apple, Not the Sofa: Impact of Irrelevant Context in Embodied AI Commands on VLA Modelscited by 4
  28. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  29. 2026Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Datasetcited by 4
  30. 2026Scaling Self-Play for End-to-End Drivingcited by 4
  31. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Surveycited by 3
  32. 2025What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectivescited by 3
  33. 2026Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Modelscited by 3
  34. 2026A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous drivingcited by 3
  35. 2026V-RAE: Rethinking Video Latent Spaces for Generationcited by 3
  36. 2025System 0/1/2/3: Quad-process theory for multi-timescale embodied collective cognitive systemscited by 2
  37. 2025Automated Data Curation Using GPS & NLP to Generate Instruction-Action Pairs for Autonomous Vehicle Vision-Language Navigation Datasetscited by 2
  38. 2025MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understandingcited by 2

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Unifying Language-Action Understanding and Generation for Autonomous Driving
  2. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  3. 2025DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models
  4. 2026StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Driving
  5. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models
  6. 2026Less Language, More Latents: Annotation-Efficient VLAs for Driving
  7. 2025CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
  8. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model