Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysis
On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection
also in 1 more
Abstract (arXiv)
Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general visual understanding. However, their application to safety-critical driving scenarios remains limited by an inability to accurately perceive and reason about rare high-stakes dynamic events, such as collisions or near-collisions. To address this, we introduce a pipeline that enhances MLLM perception by fusing downsampled video frames with synchronized high-frequency telematics data (IMU and GPS) and semantic insights from specialized computer vision models. Our pipeline generates high-quality pseudo-labels, including descriptive captions and question-answer pairs, specifically designed to train MLLMs to identify and describe Safety-Critical Events (SCEs) in real-world driving footage. We show the effectiveness of our approach fine-tuning the open-source QwenVL-2.5 model via DoRA adapters: our experiments demonstrate significant improvements in identifying and explaining safety-critical events, with fewer than 50M trainable parameters and limited computational budget.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 23 references that are in the map. First posted on arXiv 21 May 2026.
Builds on
16 of 23 references.
Closest ancestors · 3
Same direction, newest first
- 2024ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understanding1 Oct
- 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
- 2023LingoQA: Visual Question Answering for Autonomous Driving21 Dec
Other references · 13
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
- 2018Textual Explanations for Self-Driving Vehiclescited by 226
- 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
- 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
- 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
- 2025Nexar Dashcam Collision Prediction Dataset and Challengecited by 21
- 2024TLD-READY: Traffic Light Detection -- Relevance Estimation and Deployment Analysiscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events
- 2025Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends
- 2025When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis
- 2025Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding
- 2025Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
- 2025CrashChat: A Multimodal Large Language Model for Multitask Traffic Crash Video Analysis
- 2025SafePLUG: Empowering Multimodal LLMs with Pixel-Level Insight and Temporal Grounding for Traffic Accident Understanding
- 2025WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving