mlsanityData to 7 Oct 2026

Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach

arXiv 2511.23311 ↗First posted 28 Nov 2025cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Large-scale Vision Language Models (LVLMs) exhibit advanced capabilities in tasks that require visual information, including object detection. These capabilities have promising applications in various industrial domains, such as autonomous driving. For example, LVLMs can generate safety-oriented descriptions of videos captured by road-facing cameras. However, ensuring comprehensive safety requires monitoring driver-facing views as well to detect risky events, such as the use of mobiles while driving. Thus, the ability to process synchronized inputs is necessary from both driver-facing and road-facing cameras. In this study, we develop models and investigate the capabilities of LVLMs by constructing a dataset and evaluating their performance on this dataset. Our experimental results demonstrate that while pre-trained LVLMs have limited effectiveness, fine-tuned LVLMs can generate accurate and safety-aware driving instructions. Nonetheless, several challenges remain, particularly in detecting subtle or complex events in the video. Our findings and error analysis provide valuable insights that can contribute to the improvement of LVLM-based systems in this domain.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 31 references that are in the map. First posted on arXiv 28 Nov 2025.

Builds on

21 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Driving4 Sep
  2. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook22 Oct

Other references · 19

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  10. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  11. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  12. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
7 more
  1. 2020Overcoming Language Priors with Self-supervised Learning for Visual Question Answeringcited by 43
  2. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  3. 2024CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenariocited by 27
  4. 2024Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinationscited by 26
  5. 2024Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQAcited by 19
  6. 2024Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Modelscited by 7
  7. 2025Tonguescape: Exploring Language Models Understanding of Vowel Articulationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events
  2. 2025Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends
  3. 2026Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysis
  4. 2026VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events
  5. 2025Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding
  6. 2026A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions
  7. 2024Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing
  8. 2025Large Language Models for Crash Detection in Video: A Survey of Methods, Datasets, and Challenges