mlsanityData to 7 Oct 2026

ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understanding

arXiv 2410.00982 ↗First posted 1 Oct 2024cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Accurately identifying, understanding and describing traffic safety-critical events (SCEs), including crashes, tire strikes, and near-crashes, is crucial for advanced driver assistance systems, automated driving systems, and traffic safety. As SCEs are rare events, most general vision-language models (VLMs) have not been trained sufficiently to link SCE videos and narratives, which could lead to hallucinations and missing key safety characteristics. Here, we introduce ScVLM, a novel hybrid methodology that integrates supervised and contrastive learning techniques to classify the severity and types of SCEs, as well as to generate narrative descriptions of SCEs. This approach utilizes classification to enhance VLMs' comprehension of driving videos and improve the rationality of event descriptions. The proposed approach is trained on and evaluated by more than 8,600 SCEs from the Second Strategic Highway Research Program Naturalistic Driving Study dataset, the largest publicly accessible driving dataset with videos and SCE annotations. The results demonstrate the superiority of the proposed approach in generating contextually accurate event descriptions and mitigating VLM hallucinations. The code will be available at https://github.com/datadrivenwheels/ScVLM.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 45 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

26 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  2. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook22 Oct

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2021Video Swin Transformercited by 1,090
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
12 more
  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  3. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  4. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  5. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  6. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  7. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  8. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  9. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  10. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 52
  11. 2023Re-Reading Improves Reasoning in Large Language Modelscited by 37
  12. 2024Enhancing Traffic Safety with Parallel Dense Video Captioning for End-to-End Event Analysiscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understandingcited by 13
  2. 2025Better Safe Than Sorry? Overreaction Problem of Vision Language Models in Visual Emergency Recognitioncited by 7
  3. 2025Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidancecited by 3
  4. 2025Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysiscited by 3
  5. 2025Case-based Reasoning Augmented Large Language Model Framework for Decision Making in Realistic Safety-Critical Driving Scenarioscited by 2
  6. 2025Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trendscited by 1
  7. 2025CrashChat: A Multimodal Large Language Model for Multitask Traffic Crash Video Analysiscited by 1
  8. 2026D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Modelscited by 1
  9. 2025Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Drivingcited by 0
  10. 2026Towards Safer Mobile Agents: Scalable Generation and Evaluation of Diverse Scenarios for VLMscited by 0
  11. 2026Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysiscited by 0
  12. 2026Multi-modal Rail Crossing Safety Analysiscited by 0
1 more
  1. 2026DispatchRAG: Grounding Emergency Dispatch Decisions in Real-World Protocols from Traffic Accident Videocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events
  2. 2025Evaluation of Safety Cognition Capability in Vision-Language Models for Autonomous Driving
  3. 2025Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
  4. 2026VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events
  5. 2025Hierarchical Reasoning with Vision-Language Models for Incident Reports from Dashcam Videos
  6. 2025VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understanding
  7. 2026CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning
  8. 2026Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning