mlsanityData to 7 Oct 2026

Multi-Frame Vision-Language Model for Long-form Reasoning in Driver Behavior Analysis

arXiv 2408.01682 ↗First posted 3 Aug 2024cs.CL cs.CVcited by 2

On the map: LLM reasoning → Chain-of-thought reasoning for driving

also in 1 more
Abstract (arXiv)

Identifying risky driving behavior in real-world situations is essential for the safety of both drivers and pedestrians. However, integrating natural language models in this field remains relatively untapped. To address this, we created a novel multi-modal instruction tuning dataset and driver coaching inference system. Our primary use case is dashcam-based coaching for commercial drivers. The North American Dashcam Market is expected to register a CAGR of 15.4 percent from 2022 to 2027. Our dataset enables language models to learn visual instructions across various risky driving scenarios, emphasizing detailed reasoning crucial for effective driver coaching and managerial comprehension. Our model is trained on road-facing and driver-facing RGB camera footage, capturing the comprehensive scope of driving behavior in vehicles equipped with dashcams.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 9 of its 21 references that are in the map. First posted on arXiv 3 Aug 2024.

Builds on

9 of 21 references.

References · 9

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  4. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  5. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  6. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  7. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  8. 2024Towards Artwork Explanation in Large-scale Vision Language Modelscited by 8
  9. 2024IRR: Image Review Ranking Framework for Evaluating Vision-Language Modelscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Reviewcited by 8
  2. 2025Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
  2. 2026VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events
  3. 2025Hierarchical Reasoning with Vision-Language Models for Incident Reports from Dashcam Videos
  4. 2025PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behavior
  5. 2026AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding
  6. 2025Understanding Driving Risks using Large Language Models: Toward Elderly Driver Assessment
  7. 2025Case-based Reasoning Augmented Large Language Model Framework for Decision Making in Realistic Safety-Critical Driving Scenarios
  8. 2025TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos