# ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understanding

arXiv: https://arxiv.org/abs/2410.00982 · HTML: https://mlsanity.com/paper/2410.00982/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models](https://mlsanity.com/paper/2401.01313/) (2024)
- [Multi-Modal Hallucination Control by Visual Information Grounding](https://mlsanity.com/paper/2403.14003/) (2024)
- [CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP](https://mlsanity.com/paper/2301.04926/) (2023)
- [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023)
- [Large Language Models Are Not Strong Abstract Reasoners](https://mlsanity.com/paper/2305.19555/) (2023)
- [Re-Reading Improves Reasoning in Large Language Models](https://mlsanity.com/paper/2309.06275/) (2023)
- [Enhancing Traffic Safety with Parallel Dense Video Captioning for End-to-End Event Analysis](https://mlsanity.com/paper/2404.08229/) (2024)

## Built on by (most cited first, 13 of 13)

- [Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding](https://mlsanity.com/paper/2504.14526/) (2025)
- [Better Safe Than Sorry? Overreaction Problem of Vision Language Models in Visual Emergency Recognition](https://mlsanity.com/paper/2505.15367/) (2025)
- [Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance](https://mlsanity.com/paper/2508.01057/) (2025)
- [Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysis](https://mlsanity.com/paper/2510.11907/) (2025)
- [Case-based Reasoning Augmented Large Language Model Framework for Decision Making in Realistic Safety-Critical Driving Scenarios](https://mlsanity.com/paper/2506.20531/) (2025)
- [Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends](https://mlsanity.com/paper/2504.16134/) (2025)
- [CrashChat: A Multimodal Large Language Model for Multitask Traffic Crash Video Analysis](https://mlsanity.com/paper/2512.18878/) (2025)
- [Towards Safer Mobile Agents: Scalable Generation and Evaluation of Diverse Scenarios for VLMs](https://mlsanity.com/paper/2601.08470/) (2026)
- [D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models](https://mlsanity.com/paper/2607.19528/) (2026)
- [Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Driving](https://mlsanity.com/paper/2510.01126/) (2025)
- [Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysis](https://mlsanity.com/paper/2605.22185/) (2026)
- [Multi-modal Rail Crossing Safety Analysis](https://mlsanity.com/paper/2607.01365/) (2026)
- [DispatchRAG: Grounding Emergency Dispatch Decisions in Real-World Protocols from Traffic Accident Video](https://mlsanity.com/paper/2607.23132/) (2026)

## Abstract (arXiv)

Accurately identifying, understanding and describing traffic safety-critical events (SCEs), including crashes, tire strikes, and near-crashes, is crucial for advanced driver assistance systems, automated driving systems, and traffic safety. As SCEs are rare events, most general vision-language models (VLMs) have not been trained sufficiently to link SCE videos and narratives, which could lead to hallucinations and missing key safety characteristics. Here, we introduce ScVLM, a novel hybrid methodology that integrates supervised and contrastive learning techniques to classify the severity and types of SCEs, as well as to generate narrative descriptions of SCEs. This approach utilizes classification to enhance VLMs' comprehension of driving videos and improve the rationality of event descriptions. The proposed approach is trained on and evaluated by more than 8,600 SCEs from the Second Strategic Highway Research Program Naturalistic Driving Study dataset, the largest publicly accessible driving dataset with videos and SCE annotations. The results demonstrate the superiority of the proposed approach in generating contextually accurate event descriptions and mitigating VLM hallucinations. The code will be available at https://github.com/datadrivenwheels/ScVLM.