mlsanityData to 7 Oct 2026

DRAMA: Joint Risk Localization and Captioning in Driving

arXiv 2209.10767 ↗First posted 22 Sep 2022cs.CV cs.AI cs.LGcited by 125

Not on the map. Why?

Cited by 67 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Considering the functionality of situational awareness in safety-critical automation systems, the perception of risk in driving scenes and its explainability is of particular importance for autonomous and cooperative driving. Toward this goal, this paper proposes a new research direction of joint risk localization in driving scenes and its risk explanation as a natural language description. Due to the lack of standard benchmarks, we collected a large-scale dataset, DRAMA (Driving Risk Assessment Mechanism with A captioning module), which consists of 17,785 interactive driving scenarios collected in Tokyo, Japan. Our DRAMA dataset accommodates video- and object-level questions on driving risks with associated important objects to achieve the goal of visual captioning as a free-form language description utilizing closed and open-ended responses for multi-level questions, which can be used to evaluate a range of visual captioning capabilities in driving scenarios. We make this data available to the community for further research. Using DRAMA, we explore multiple facets of joint risk localization and captioning in interactive driving scenarios. In particular, we benchmark various multi-task prediction architectures and provide a detailed analysis of joint risk localization and risk captioning. The data set is available at https://usa.honda-ri.com/drama

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 22 Sep 2022.

Builds on

32 of 44 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2015Fast R-CNNcited by 2,930
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  11. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  12. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
20 more
  1. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  2. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  3. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  4. 2015Natural Language Object Retrievalcited by 124
  5. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  6. 2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 106
  7. 2018Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoningcited by 102
  8. 2020Explainable Object-induced Action Decision for Autonomous Vehiclescited by 88
  9. 2019Grounding Human-to-Vehicle Advice for Self-driving Vehiclescited by 71
  10. 2017Predicting Driver Attention in Critical Situationscited by 61
  11. 2020TITAN: Future Forecast using Action Priorscited by 58
  12. 2018Deep Object-Centric Policies for Autonomous Drivingcited by 41
  13. 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
  14. 2017Query-guided Regression Network with Context Policy for Phrase Groundingcited by 34
  15. 2020Who Make Drivers Stop? Towards Driver-centric Risk Assessment: Risk Object Identification via Causal Inferencecited by 34
  16. 2017Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localizationcited by 33
  17. 2019Context and Attribute Grounded Dense Captioningcited by 31
  18. 2020When, Where, and What? A New Dataset for Anomaly Detection in Driving Videoscited by 27
  19. 2016Top-down Visual Saliency Guided by Captionscited by 22
  20. 2019Goal-oriented Object Importance Estimation in On-road Driving Videoscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 17; 2024: 31; 2025: 47; 2026: 29.

125 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  5. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  6. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  7. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  8. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  9. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  10. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  11. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  12. 2023Language Prompt for Autonomous Drivingcited by 110
38 more
  1. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  2. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  3. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  4. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  5. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  6. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  7. 2024Embodied Understanding of Driving Scenarioscited by 63
  8. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  9. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  10. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  11. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  12. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  13. 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33
  14. 2023Vision-Based Traffic Accident Detection and Anticipation: A Surveycited by 31
  15. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotationscited by 31
  16. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  17. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  18. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  19. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  20. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  21. 2024Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Drivingcited by 22
  22. 2025VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understandingcited by 20
  23. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  24. 2024VisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptionscited by 18
  25. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  26. 2024WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understandingcited by 17
  27. 2025DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Modelscited by 17
  28. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  29. 2022Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmarkcited by 16
  30. 2025Embodied Scene Understanding for Vision Language Models via MetaVQAcited by 16
  31. 2024Driving Style Alignment for LLM-powered Driver Agentcited by 15
  32. 2025RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narrativescited by 15
  33. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigationcited by 14
  34. 2025Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directionscited by 14
  35. 2025SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generationcited by 14
  36. 2024IDD-X: A Multi-View Dataset for Ego-relative Important Object Localization and Explanation in Dense and Unstructured Trafficcited by 13
  37. 2025Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understandingcited by 13
  38. 2023RiskBench: A Scenario-based Benchmark for Risk Identificationcited by 12

50 most cited of 125 All 125 (JSON)