mlsanityData to 14 Oct 2026

Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning

arXiv 2309.06597 ↗First posted 12 Sep 2023cs.CV cs.AI cs.LGcited by 63

Not on the map. Why?

Cited by 36 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The widespread adoption of commercial autonomous vehicles (AVs) and advanced driver assistance systems (ADAS) may largely depend on their acceptance by society, for which their perceived trustworthiness and interpretability to riders are crucial. In general, this task is challenging because modern autonomous systems software relies heavily on black-box artificial intelligence models. Towards this goal, this paper introduces a novel dataset, Rank2Tell, a multi-modal ego-centric dataset for Ranking the importance level and Telling the reason for the importance. Using various close and open-ended visual question answering, the dataset provides dense annotations of various semantic, spatial, temporal, and relational attributes of various important objects in complex traffic scenarios. The dense annotations and unique attributes of the dataset make it a valuable resource for researchers working on visual scene understanding and related fields. Furthermore, we introduce a joint model for joint importance level ranking and natural language captions generation to benchmark our dataset and demonstrate performance with quantitative evaluations.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 35 of its 53 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

35 of 53 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  11. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  12. 2018Textual Explanations for Self-Driving Vehiclescited by 226
23 more
  1. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  2. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  3. 2020Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentationcited by 144
  4. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 142
  5. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  6. 2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 107
  7. 2018Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoningcited by 102
  8. 2017Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Imagescited by 101
  9. 2020Explainable Object-induced Action Decision for Autonomous Vehiclescited by 88
  10. 2017Towards Diverse and Natural Image Descriptions via a Conditional GANcited by 59
  11. 2019Describing like humans: on diversity in image captioningcited by 46
  12. 2019VoteNet: A Deep Learning Label Fusion Method for Multi-Atlas Segmentationcited by 44
  13. 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
  14. 2020Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationshipscited by 37
  15. 2021RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecastingcited by 36
  16. 2020Who Make Drivers Stop? Towards Driver-centric Risk Assessment: Risk Object Identification via Causal Inferencecited by 34
  17. 2017Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localizationcited by 33
  18. 2019Context and Attribute Grounded Dense Captioningcited by 31
  19. 2020When, Where, and What? A New Dataset for Anomaly Detection in Driving Videoscited by 27
  20. 2019Goal-oriented Object Importance Estimation in On-road Driving Videoscited by 19
  21. 2020Interaction Graphs for Object Importance Estimation in On-road Driving Videoscited by 15
  22. 2022Important Object Identification with Semi-Supervised Learning for Autonomous Drivingcited by 13
  23. 2019Learning to Learn Relation for Important People Detection in Still Imagescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 17; 2025: 28; 2026: 9.

63 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  3. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  4. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  5. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 145
  6. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 143
  7. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  8. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  9. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 123
  10. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
  11. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  12. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 66
38 more
  1. 2024Embodied Understanding of Driving Scenarioscited by 63
  2. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  4. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 41
  5. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  6. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  7. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  8. 2025When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysiscited by 21
  9. 2025VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understandingcited by 20
  10. 2025Embodied Scene Understanding for Vision Language Models via MetaVQAcited by 19
  11. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  12. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  13. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  14. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigationcited by 15
  15. 2025RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narrativescited by 15
  16. 2025Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directionscited by 14
  17. 2025Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understandingcited by 13
  18. 2024Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Mapcited by 11
  19. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  20. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  21. 2025DriveQA: Passing the Driving Knowledge Testcited by 10
  22. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 10
  23. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  24. 2024Disentangled Neural Relational Inference for Interpretable Motion Predictioncited by 7
  25. 2025V3LMA: Visual 3D-enhanced Language Model for Autonomous Drivingcited by 7
  26. 2025A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Drivingcited by 5
  27. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Modelscited by 5
  28. 2025CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Drivingcited by 5
  29. 2023Exploring the Potential of Multi-Modal AI for Driving Hazard Predictioncited by 4
  30. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 4
  31. 2025PRIMEDrive-CoT: A Precognitive Chain-of-Thought Framework for Uncertainty-Aware Object Interaction in Driving Scene Scenariocited by 3
  32. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Drivingcited by 3
  33. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 3
  34. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  35. 2024Potential Field as Scene Affordance for Behavior Change-Based Visual Risk Object Identificationcited by 2
  36. 2025TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videoscited by 2
  37. 2025From Static to Dynamic: a Survey of Topology-Aware Perception in Autonomous Drivingcited by 2
  38. 2025From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Modelscited by 2

50 most cited of 63 All 63 (JSON)