mlsanityData to 7 Oct 2026

OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning

arXiv 2504.04348 ↗First posted 6 Apr 2025cs.CVcited by 142

On the map: World models → World models for autonomous driving

also in 3 more
Abstract (arXiv)

The advances in vision-language models (VLMs) have led to a growing interest in autonomous driving to leverage their strong reasoning capabilities. However, extending these capabilities from 2D to full 3D understanding is crucial for real-world applications. To address this challenge, we propose OmniDrive, a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning. This approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions. Our counterfactual-based synthetic data annotation process generates large-scale, high-quality datasets, providing denser supervision signals that bridge planning trajectories and language-based reasoning. Futher, we explore two advanced OmniDrive-Agent frameworks, namely Omni-L and Omni-Q, to assess the importance of vision-language alignment versus 3D perception, revealing critical insights into designing effective LLM-agents. Significant improvements on the DriveLM Q&A benchmark and nuScenes open-loop planning demonstrate the effectiveness of our dataset and methods.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 51 of its 59 references that are in the map. First posted on arXiv 6 Apr 2025.

Builds on

51 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
38 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2022Planning-oriented Autonomous Drivingcited by 1,068
  3. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  5. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  6. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  7. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  8. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  9. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  10. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  11. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  12. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  13. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  14. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  15. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  16. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  17. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  18. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  19. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  20. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  21. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  22. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  23. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  24. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  25. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  26. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 177
  27. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  28. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  29. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  30. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  31. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  32. 2023Language Prompt for Autonomous Drivingcited by 110
  33. 2023OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mappingcited by 83
  34. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  35. 2019Grounding Human-to-Vehicle Advice for Self-driving Vehiclescited by 71
  36. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  37. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  38. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 40; 2026: 102.

142 papers.

Sort:

By area:

  1. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  2. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  3. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 38
  4. 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Drivingcited by 35
  5. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  6. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  7. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  8. 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Drivingcited by 26
  9. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  10. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 24
  11. 2025Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Drivingcited by 24
  12. 2025Embodied AI: From LLMs to World Modelscited by 23
38 more
  1. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  2. 2025FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruningcited by 20
  3. 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Drivingcited by 20
  4. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  5. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 18
  6. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 17
  7. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 17
  8. 2025OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervisioncited by 16
  9. 2025Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Drivingcited by 15
  10. 2025WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Drivingcited by 15
  11. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  12. 2025Latent Chain-of-Thought World Modeling for End-to-End Drivingcited by 14
  13. 2025MindDrive: An All-in-One Framework Bridging World Models and Vision-Language Model for End-to-End Autonomous Drivingcited by 12
  14. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 12
  15. 2026DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Drivingcited by 12
  16. 2026DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Drivingcited by 11
  17. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  18. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  19. 2026CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Drivingcited by 10
  20. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 10
  21. 2025CHAI: Command Hijacking against embodied AIcited by 9
  22. 2026Unifying Language-Action Understanding and Generation for Autonomous Drivingcited by 9
  23. 2025RoCA: Robust Cross-Domain End-to-End Autonomous Drivingcited by 8
  24. 2025Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Drivingcited by 8
  25. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 8
  26. 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Drivingcited by 8
  27. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 8
  28. 2026CausalVAD: De-confounding End-to-End Autonomous Driving via Causal Interventioncited by 7
  29. 2025ALN-P3: Unified Language Alignment for Perception, Prediction, and Planning in Autonomous Drivingcited by 6
  30. 2026Vega: Learning to Drive with Natural Language Instructionscited by 6
  31. 2026MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Drivingcited by 6
  32. 2026What-If World: A Causal Benchmark for General World Models in Embodied Scenarioscited by 6
  33. 2025CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Drivingcited by 5
  34. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  35. 2026SparseOccVLA: Bridging Occupancy and Vision-Language Models via Sparse Queries for Unified 4D Scene Understanding and Planningcited by 4
  36. 2026RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Spacecited by 4
  37. 2026More than the Sum: Panorama-Language Models for Adverse Omni-Scenescited by 4
  38. 2026Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planningcited by 4

50 most cited of 142 All 142 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning
  2. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
  3. 2025OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving
  4. 2025Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoning
  5. 2025DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving
  6. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving
  7. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving
  8. 2025CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving