mlsanityData to 7 Oct 2026

DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding

arXiv 2603.11380 ↗First posted 11 Mar 2026cs.CVcited by 0

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

Fusing sensors with complementary modalities is crucial for maintaining a stable and comprehensive understanding of abnormal driving scenes. However, Multimodal Large Language Models (MLLMs) are underexplored for leveraging multi-sensor information to understand adverse driving scenarios in autonomous vehicles. To address this gap, we propose the DriveXQA, a multimodal dataset for autonomous driving VQA. In addition to four visual modalities, five sensor failure cases, and five weather conditions, it includes 102,505 QA pairs categorized into three types: global scene level, allocentric level, and ego-vehicle centric level. Since no existing MLLM framework adopts multiple complementary visual modalities as input, we design MVX-LLM, a token-efficient architecture with a Dual Cross-Attention (DCA) projector that fuses the modalities to alleviate information redundancy. Experiments demonstrate that our DCA achieves improved performance under challenging conditions such as foggy (GPTScore: 53.5 vs. 25.1 for the baseline).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 43 of its 51 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

43 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models17 Mar
  2. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  3. 2023LingoQA: Visual Question Answering for Autonomous Driving21 Dec
  4. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations11 Dec
  5. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving3 Oct

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  12. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
26 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  5. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  6. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  7. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  8. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  9. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  10. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  11. 2022Multimodal Token Fusion for Vision Transformerscited by 132
  12. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  13. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  14. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 115
  15. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 58
  16. 2024Gramian Multimodal Representation Learning and Alignmentcited by 52
  17. 2024What to align in multimodal contrastive learning?cited by 41
  18. 2024ParGo: Bridging Vision-Language with Partial and Global Viewscited by 40
  19. 2022Multi-modal Contrastive Representation Learning for Entity Alignmentcited by 37
  20. 2024CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenescited by 33
  21. 2025DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentationcited by 30
  22. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  23. 2022Early or Late Fusion Matters: Efficient RGB-D Fusion in Vision Transformers for 3D Object Recognitioncited by 11
  24. 2024Rethinking Early-Fusion Strategies for Improved Multispectral Object Detectioncited by 9
  25. 2023Towards a performance analysis on pre-trained Visual Question Answering models for autonomous drivingcited by 3
  26. 2024Multi-modal Semantic Understanding with Contrastive Cross-modal Feature Alignmentcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
  2. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding
  3. 2026V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
  4. 2025Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding
  5. 2025Hierarchical Question-Answering for Driving Scene Understanding Using Vision-Language Models
  6. 2026AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding
  7. 2024Application of Multimodal Large Language Models in Autonomous Driving
  8. 2025AutoDrive-QA: A Multiple-Choice Benchmark for Vision-Language Evaluation in Urban Autonomous Driving