mlsanityData to 14 Oct 2026

DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models

arXiv 2505.07084 ↗First posted 11 May 2025cs.ROcited by 4

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Human drivers possess spatial and causal intelligence, enabling them to perceive driving scenarios, anticipate hazards, and react to dynamic environments. In contrast, autonomous vehicles lack these abilities, making it challenging to manage perception-related Safety of the Intended Functionality (SOTIF) risks, especially under complex or unpredictable driving conditions. To address this gap, we propose fine-tuning multimodal large language models (MLLMs) on a customized dataset specifically designed to capture perception-related SOTIF scenarios. Benchmarking results show that fine-tuned MLLMs achieve an 11.8% improvement in close-ended VQA accuracy and a 12.0% increase in open-ended VQA scores compared to baseline models, while maintaining real-time performance with a 0.59-second average inference time per image. We validate our approach through real-world case studies in Canada and China, where fine-tuned models correctly identify safety risks that challenge even experienced human drivers. This work represents the first application of domain-specific MLLM fine-tuning for SOTIF domain in autonomous driving. The dataset and related resources are available at github.com/s95huang/DriveSOTIF.git

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 61 references that are in the map. First posted on arXiv 11 May 2025.

Builds on

42 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases16 Apr
  2. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving28 Mar

Other references · 40

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
28 more
  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  4. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  5. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  8. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  9. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  10. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  11. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  12. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  13. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  14. 2023Referring Multi-Object Trackingcited by 93
  15. 2020Canadian Adverse Driving Conditions Datasetcited by 87
  16. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 79
  17. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 59
  18. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 34
  19. 2024VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Drivingcited by 22
  20. 2023Explaining Autonomous Driving Actions with Visual Question Answeringcited by 16
  21. 2022PeSOTIF: a Challenging Visual Dataset for Perception SOTIF Problems in Long-tail Traffic Scenarioscited by 8
  22. 2023Context-VQA: Towards Context-Aware and Purposeful Visual Question Answeringcited by 6
  23. 2022SOTIF Entropy: Online SOTIF Risk Quantification and Mitigation for Autonomous Drivingcited by 5
  24. 2022Formal Certification Methods for Automated Vehicle Safety Assessmentcited by 4
  25. 2023Systematic Modeling Approach for Environmental Perception Limitations in Automated Drivingcited by 2
  26. 2025Decomposition and Quantification of SOTIF Requirements for Perception Systems of Autonomous Vehiclescited by 2
  27. 2022A Holistic Robust Motion Controller Framework for Autonomous Platooningcited by 1
  28. 2025Simulation-Based Performance Evaluation of 3D Object Detection Methods with Deep Learning for a LiDAR Point Cloud Dataset in a SOTIF-related Use Casecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Seeing before Observable: Potential Risk Reasoning in Autonomous Driving via Vision Language Modelscited by 2
  2. 2026Listen, Look, Drive: Coupling Audio Instructions for User-aware VLA-based Autonomous Drivingcited by 2
  3. 2026A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditionscited by 1
  4. 2026Large Multimodal Models for Embodied Intelligent Driving: The Next Frontier in Self-Driving?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends
  2. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
  3. 2026V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
  4. 2025Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding
  5. 2024Probing Multimodal LLMs as World Models for Driving
  6. 2026Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysis
  7. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving
  8. 2026UniDrive: A Unified Vision-Language and Grounding Framework for Interpretable Risk Understanding in Autonomous Driving