mlsanityData to 7 Oct 2026

OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning

arXiv 2405.01533 ↗First posted 2 May 2024cs.CVcited by 26

On the map: LLM reasoning → Chain-of-thought reasoning for driving

also in 2 more
Abstract (arXiv)

The advances in vision-language models (VLMs) have led to a growing interest in autonomous driving to leverage their strong reasoning capabilities. However, extending these capabilities from 2D to full 3D understanding is crucial for real-world applications. To address this challenge, we propose OmniDrive, a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning. This approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions. Our counterfactual-based synthetic data annotation process generates large-scale, high-quality datasets, providing denser supervision signals that bridge planning trajectories and language-based reasoning. Futher, we explore two advanced OmniDrive-Agent frameworks, namely Omni-L and Omni-Q, to assess the importance of vision-language alignment versus 3D perception, revealing critical insights into designing effective LLM-agents. Significant improvements on the DriveLM Q&A benchmark and nuScenes open-loop planning demonstrate the effectiveness of our dataset and methods.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 27 of its 56 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

27 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving6 Dec

Other references · 26

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  7. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  8. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  9. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  10. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  11. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  12. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
14 more
  1. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  2. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  3. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  4. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  5. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  6. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  7. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  8. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  9. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 177
  10. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  11. 2023Language Prompt for Autonomous Drivingcited by 110
  12. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  13. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  14. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45

Built on by

Cited by year

Citing papers per year: 2024: 8; 2025: 18.

26 papers.

Sort:

By area:

  1. 20253D and 4D World Modeling: A Surveycited by 64
  2. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  3. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learningcited by 38
  4. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  5. 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Drivingcited by 35
  6. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Drivingcited by 34
  7. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Taskscited by 27
  8. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoningcited by 27
  9. 2024ViLLa: Video Reasoning Segmentation with Large Language Modelcited by 23
  10. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  11. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Drivingcited by 19
  12. 2025Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Drivingcited by 16
14 more
  1. 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Drivingcited by 15
  2. 2025Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Drivingcited by 14
  3. 2025ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Modelscited by 13
  4. 2024World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Drivingcited by 9
  5. 2025RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Drivingcited by 8
  6. 2025Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMscited by 6
  7. 2025TS-VLM: Text-Guided SoftSort Pooling for Vision-Language Models in Multi-View Driving Reasoningcited by 5
  8. 2025DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Drivingcited by 5
  9. 2025NeSyPr: Neurosymbolic Proceduralization For Efficient Embodied Reasoningcited by 5
  10. 2024Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagerycited by 4
  11. 2025MoSE: Skill-by-Skill Mixture-of-Experts Learning for Embodied Autonomous Machinescited by 3
  12. 2025iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoningcited by 3
  13. 2025Universal Camouflage Attack on Vision-Language Models for Autonomous Drivingcited by 3
  14. 2024A Novel MLLM-based Approach for Autonomous Driving in Different Weather Conditionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning
  2. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
  3. 2025OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving
  4. 2025Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoning
  5. 2026DriveReward: A Comprehensive Dataset and Generative Vision-Language Reward Model for Autonomous Driving
  6. 2026C-CoT: Counterfactual Chain-of-Thought with Vision-Language Models for Safe Autonomous Driving
  7. 2025ALN-P3: Unified Language Alignment for Perception, Prediction, and Planning in Autonomous Driving
  8. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving