mlsanityData to 7 Oct 2026

WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving

arXiv 2407.04281 ↗First posted 5 Jul 2024cs.ROcited by 19

On the map: LLM reasoning → Chain-of-thought reasoning for driving

also in 1 more
Abstract (arXiv)

Language models uncover unprecedented abilities in analyzing driving scenarios, owing to their limitless knowledge accumulated from text-based pre-training. Naturally, they should particularly excel in analyzing rule-based interactions, such as those triggered by traffic laws, which are well documented in texts. However, such interaction analysis remains underexplored due to the lack of dedicated language datasets that address it. Therefore, we propose Waymo Open Motion Dataset-Reasoning (WOMD-Reasoning), a comprehensive large-scale Q&As dataset built on WOMD focusing on describing and reasoning traffic rule-induced interactions in driving scenarios. WOMD-Reasoning also presents by far the largest multi-modal Q&A dataset, with 3 million Q&As on real-world driving scenarios, covering a wide range of driving topics from map descriptions and motion status descriptions to narratives and analyses of agents' interactions, behaviors, and intentions. To showcase the applications of WOMD-Reasoning, we design Motion-LLaVA, a motion-language model fine-tuned on WOMD-Reasoning. Quantitative and qualitative evaluations are performed on WOMD-Reasoning dataset as well as the outputs of Motion-LLaVA, supporting the data quality and wide applications of WOMD-Reasoning, in interaction predictions, traffic rule compliance plannings, etc. The dataset and its vision modal extension are available on https://waymo.com/open/download/. The codes & prompts to build it are available on https://github.com/yhli123/WOMD-Reasoning.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 22 of its 75 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

22 of 75 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning2 May
  2. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving6 Dec

Other references · 20

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  3. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  4. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  5. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  6. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  7. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  8. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  9. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  10. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  11. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  12. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
8 more
  1. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  2. 2024Embodied Understanding of Driving Scenarioscited by 63
  3. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 55
  4. 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Drivingcited by 41
  5. 2023WOMD-LiDAR: Raw Sensor Dataset Benchmark for Motion Forecastingcited by 28
  6. 2024Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Setscited by 25
  7. 2024MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationalecited by 14
  8. 2022MPA: MultiPath++ Based Architecture for Motion Predictioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 11.

19 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  3. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  4. 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospectscited by 12
  5. 2025TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Predictioncited by 9
  6. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  7. 2026BridgeSim: Unveiling the OL-CL Gap in End-to-End Autonomous Drivingcited by 9
  8. 2026Accelerating Structured Chain-of-Thought in Autonomous Vehiclescited by 6
  9. 2025DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Drivingcited by 5
  10. 2026A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigmscited by 1
  11. 2025Multiscale Video Transformers for Class Agnostic Segmentation in Autonomous Drivingcited by 0
  12. 2026GenPlanner: From Noise to Plans -- Emergent Reasoning in Flow Matching and Diffusion Modelscited by 0
7 more
  1. 2026General-purpose LLMs as Models of Human Driver Behavior: The Case of Simplified Mergingcited by 0
  2. 2026What Probing Reveals about Autonomous Driving: Linking Internal Prediction Errors to Ego Planningcited by 0
  3. 2026Agent-driven Long-tail Simulation for Autonomous Drivingcited by 0
  4. 2026Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMscited by 0
  5. 2026Exploring LLM Capabilities for Situational Understanding and COLREG compliance on real-world maritime navigation scenarioscited by 0
  6. 2026NavSafe-∞: Benchmarking Closed-Loop Driving Safety in Photorealistic Environmentscited by 0
  7. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps
  2. 2026DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving
  3. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Models
  4. 2026CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving
  5. 2026Large Language Model based Interactive Decision-Making for Autonomous Driving
  6. 2026An interactive enhanced driving dataset for autonomous driving
  7. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding
  8. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving