mlsanityData to 7 Oct 2026

Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning

arXiv 1811.02307 ↗First posted 6 Nov 2018cs.CVcited by 102

Not on the map. Why?

Abstract (arXiv)

Driving Scene understanding is a key ingredient for intelligent transportation systems. To achieve systems that can operate in a complex physical and social environment, they need to understand and learn how humans drive and interact with traffic scenes. We present the Honda Research Institute Driving Dataset (HDD), a challenging dataset to enable research on learning driver behavior in real-life environments. The dataset includes 104 hours of real human driving in the San Francisco Bay Area collected using an instrumented vehicle equipped with different sensors. We provide a detailed analysis of HDD with a comparison to other driving datasets. A novel annotation methodology is introduced to enable research on driver behavior understanding from untrimmed data sequences. As the first step, baseline algorithms for driver behavior detection are trained and tested to demonstrate the feasibility of the proposed task.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 17; 2022: 16; 2023: 22; 2024: 19; 2025: 15; 2026: 12.

102 papers.

Sort:

By area:

  1. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  2. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  3. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  4. 2021OadTR: Online Action Detection with Transformerscited by 83
  5. 2023OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mappingcited by 83
  6. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  7. 2021Explanations in Autonomous Driving: A Surveycited by 80
  8. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  9. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  10. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  11. 2022Multi-modal Sensor Fusion for Auto Driving Perception: A Surveycited by 73
  12. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
38 more
  1. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  2. 2021ROAD: The ROad event Awareness Dataset for Autonomous Drivingcited by 54
  3. 2024Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Reviewcited by 52
  4. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  5. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  6. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  7. 2023GaitGCI: Generative Counterfactual Intervention for Gait Recognitioncited by 30
  8. 2019MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learningcited by 29
  9. 2021Spatio-Temporal Scene-Graph Embedding for Autonomous Vehicle Collision Predictioncited by 29
  10. 2023Data-driven Traffic Simulation: A Comprehensive Reviewcited by 29
  11. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  12. 2024SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generationcited by 28
  13. 2022GateHUB: Gated History Unit with Background Suppression for Online Action Detectioncited by 27
  14. 2022MUAD: Multiple Uncertainties for Autonomous Driving, a benchmark for multiple uncertainty types and taskscited by 24
  15. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  16. 2021CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoningcited by 23
  17. 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 22
  18. 2022Colar: Effective and Efficient Online Action Detection by Consulting Exemplarscited by 21
  19. 2022CausalAgents: A Robustness Benchmark for Motion Forecasting using Causal Relationshipscited by 18
  20. 2023RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understandingcited by 17
  21. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  22. 2021roadscene2vec: A Tool for Extracting and Embedding Road Scene-Graphscited by 15
  23. 2025RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narrativescited by 15
  24. 2021Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognitioncited by 14
  25. 2024IDD-X: A Multi-View Dataset for Ego-relative Important Object Localization and Explanation in Dense and Unstructured Trafficcited by 13
  26. 2023InfraParis: A multi-modal and multi-task autonomous driving datasetcited by 12
  27. 2023Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenescited by 12
  28. 2023RiskBench: A Scenario-based Benchmark for Risk Identificationcited by 12
  29. 2024Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigmcited by 12
  30. 2021METEOR:A Dense, Heterogeneous, and Unstructured Traffic Dataset With Rare Behaviorscited by 11
  31. 2022Estimating counterfactual treatment outcomes over time in complex multiagent scenarioscited by 11
  32. 2024CELLO: Causal Evaluation of Large Vision-Language Modelscited by 10
  33. 2021Behavioral Research and Practical Models of Drivers' Attentioncited by 9
  34. 2021DROID: Driver-centric Risk Object Identificationcited by 9
  35. 2023Explainable Action Prediction through Self-Supervision on Scene Graphscited by 9
  36. 2022From Spoken Thoughts to Automated Driving Commentary: Predicting and Explaining Intelligent Vehicles' Actionscited by 8
  37. 2023Ego-Vehicle Action Recognition based on Semi-Supervised Contrastive Learningcited by 8
  38. 2023E2E-LOAD: End-to-End Long-form Online Action Detectioncited by 8

50 most cited of 102 All 102 (JSON)