mlsanityData to 14 Oct 2026

Vision-Language Models can Identify Distracted Driver Behavior from Naturalistic Videos

arXiv 2306.10159 ↗First posted 16 Jun 2023cs.CVcited by 17

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Recognizing the activities causing distraction in real-world driving scenarios is critical for ensuring the safety and reliability of both drivers and pedestrians on the roadways. Conventional computer vision techniques are typically data-intensive and require a large volume of annotated training data to detect and classify various distracted driving behaviors, thereby limiting their efficiency and scalability. We aim to develop a generalized framework that showcases robust performance with access to limited or no annotated training data. Recently, vision-language models have offered large-scale visual-textual pretraining that can be adapted to task-specific learning like distracted driving activity recognition. Vision-language pretraining models, such as CLIP, have shown significant promise in learning natural language-guided visual representations. This paper proposes a CLIP-based driver activity recognition approach that identifies driver distraction from naturalistic driving images and videos. CLIP's vision embedding offers zero-shot transfer and task-based finetuning, which can classify distracted activities from driving video data. Our results show that this framework offers state-of-the-art performance on zero-shot transfer and video-based CLIP for predicting the driver's state on two public datasets. We propose both frame-based and video-based frameworks developed on top of the CLIP's visual representation for distracted driving detection and classification tasks and report the results.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 93 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

47 of 93 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  10. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  11. 2018SlowFast Networks for Video Recognitioncited by 1,867
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
35 more
  1. 2021ViViT: A Video Vision Transformercited by 1,620
  2. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  3. 2021Video Swin Transformercited by 1,091
  4. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  5. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  6. 2021Multiscale Vision Transformerscited by 897
  7. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  8. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  9. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  10. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  11. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  12. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  13. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  14. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  15. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  16. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  17. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  18. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  19. 2017Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognitioncited by 178
  20. 2020Support-set bottlenecks for video-text representation learningcited by 170
  21. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  22. 2018Dual Encoding for Zero-Example Video Retrievalcited by 85
  23. 2018D3D: Distilled 3D Networks for Video Action Recognitioncited by 60
  24. 2020DMD: A Large-Scale Multi-Modal Driver Monitoring Dataset for Attention and Alertness Analysiscited by 47
  25. 2020Driver Anomaly Detection: A Dataset and Contrastive Learning Approachcited by 41
  26. 2018Spatio-Temporal Channel Correlation Networks for Action Classificationcited by 38
  27. 2017Real-time Distracted Driver Posture Classificationcited by 29
  28. 2019Driver Distraction Identification with an Ensemble of Convolutional Neural Networkscited by 26
  29. 2022MovieCLIP: Visual Scene Recognition in Moviescited by 16
  30. 2018Real-Time Rotation-Invariant Face Detection with Progressive Calibration Networkscited by 15
  31. 2022Synthetic Distracted Driving (SynDD2) dataset for analyzing distracted behaviors and various gaze zones of a drivercited by 15
  32. 2018Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrievalcited by 10
  33. 2023Toward Extremely Lightweight Distracted Driver Recognition With Distillation-Based Neural Architecture Search and Knowledge Transfercited by 9
  34. 2019Real-Time Driver State Monitoring Using a CNN Based Spatio-Temporal Approachcited by 6
  35. 2022Is my Driver Observation Model Overconfident? Input-guided Calibration Networks for Reliable and Interpretable Confidence Estimatescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 8; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 126
  2. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  3. 2025When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysiscited by 21
  4. 2024Efficient Mixture-of-Expert for Video-based Driver State and Physiological Multi-task Estimation in Conditional Autonomous Drivingcited by 9
  5. 2025Planning Safety Trajectories with Dual-Phase, Physics-Informed, and Transportation Knowledge-Driven Large Language Modelscited by 5
  6. 2024Evaluation and Comparison of Visual Language Models for Transportation Engineering Problemscited by 4
  7. 2024PO-GUISE+: Pose and object guided transformer token selection for efficient driver action recognitioncited by 2
  8. 2025Vision-based 3D Semantic Scene Completion via Capture Dynamic Representationscited by 2
  9. 2025Agentic Vehicles for Human-Centered Mobility: Definition, Prospects, and Synergistic Co-Development with Vehicle Autonomycited by 2
  10. 2023Enhancing Cross-Dataset Performance of Distracted Driving Detection With Score Softmax Classifier And Dynamic Gaussian Smoothing Supervisioncited by 1
  11. 2025Exploring Video-Based Driver Activity Recognition under Noisy Labelscited by 1
  12. 2025PQ-DAF: Pose-driven Quality-controlled Data Augmentation for Data-scarce Driver Distraction Detectioncited by 0
5 more
  1. 2025Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Drivingcited by 0
  2. 2025C-DIRA: Computationally Efficient Dynamic ROI Routing and Domain-Invariant Adversarial Learning for Lightweight Driver Behavior Recognitioncited by 0
  3. 2026Zero-Shot Distracted Driver Detection via Vision Language Models with Double Decouplingcited by 0
  4. 2026Vision-language Models for Driver Monitoring Systems: A Driver Activity Description Datasetcited by 0
  5. 2026Improving Semantic Coverage in AI Test & Evaluation through Generative Augmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Driver Activity Classification Using Generalizable Representations from Vision-Language Models
  2. 2025CrashChat: A Multimodal Large Language Model for Multitask Traffic Crash Video Analysis
  3. 2024DSDFormer: An Innovative Transformer-Mamba Framework for Robust High-Precision Driver Distraction Identification
  4. 2025Vision-Language Models for Autonomous Driving: CLIP-Based Dynamic Scene Understanding
  5. 2024Cross-Camera Distracted Driver Classification through Feature Disentanglement and Contrastive Learning
  6. 2026Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Driving
  7. 2024Multi-Frame Vision-Language Model for Long-form Reasoning in Driver Behavior Analysis
  8. 2025DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Driving