# Driver Activity Classification Using Generalizable Representations from Vision-Language Models

arXiv: https://arxiv.org/abs/2404.14906 · HTML: https://mlsanity.com/paper/2404.14906/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 18 of 18)

- [TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration](https://mlsanity.com/paper/2203.00927/) (2022)
- [Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets](https://mlsanity.com/paper/2402.07320/) (2024)
- [The 7th AI City Challenge](https://mlsanity.com/paper/2304.07500/) (2023)
- [Navigating Open Set Scenarios for Skeleton-based Action Recognition](https://mlsanity.com/paper/2312.06330/) (2023)
- [The Why, When, and How to Use Active Learning in Large-Data-Driven 3D Object Detection for Safe Autonomous Driving: An Empirical Exploration](https://mlsanity.com/paper/2401.16634/) (2024)
- [Language-Driven Active Learning for Diverse Open-Set 3D Object Detection](https://mlsanity.com/paper/2404.12856/) (2024)
- [A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding](https://mlsanity.com/paper/2204.04734/) (2022)
- [Autonomous Vehicles that Alert Humans to Take-Over Controls: Modeling with Real-World Data](https://mlsanity.com/paper/2104.11489/) (2021)
- [Delving Deep into One-Shot Skeleton-based Action Recognition with Diverse Occlusions](https://mlsanity.com/paper/2202.11423/) (2022)
- [Safe Control Transitions: Machine Vision Based Observable Readiness Index and Data-Driven Takeover Time Prediction](https://mlsanity.com/paper/2301.05805/) (2023)
- [Informed Democracy: Voting-based Novelty Detection for Action Recognition](https://mlsanity.com/paper/1810.12819/) (2018)
- [Predicting Take-over Time for Autonomous Driving with Real-World Data: Robust Data Augmentation, Models, and Evaluation](https://mlsanity.com/paper/2107.12932/) (2021)
- [Pedestrian Behavior Maps for Safety Advisories: CHAMP Framework and Real-World Data Analysis](https://mlsanity.com/paper/2305.04506/) (2023)
- [Uncertainty-sensitive Activity Recognition: a Reliability Benchmark and the CARING Models](https://mlsanity.com/paper/2101.00468/) (2021)
- [(Safe) SMART Hands: Hand Activity Analysis and Distraction Alerts Using a Multi-Camera Framework](https://mlsanity.com/paper/2301.05838/) (2023)
- [Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognition](https://mlsanity.com/paper/2301.12592/) (2023)
- [CHAMP: Crowdsourced, History-Based Advisory of Mapped Pedestrians for Safer Driver Assistance Systems](https://mlsanity.com/paper/2301.05842/) (2023)
- [Learning to Find Missing Video Frames with Synthetic Data Augmentation: A General Framework and Application in Generating Thermal Images Using RGB Cameras](https://mlsanity.com/paper/2403.00196/) (2024)

## Built on by (most cited first, 3 of 3)

- [Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safety](https://mlsanity.com/paper/2410.12225/) (2024)
- [Beyond General Prompts: Automated Prompt Refinement using Contrastive Class Alignment Scores for Disambiguating Objects in Vision-Language Models](https://mlsanity.com/paper/2505.09139/) (2025)
- [Zero-Shot Distracted Driver Detection via Vision Language Models with Double Decoupling](https://mlsanity.com/paper/2601.08467/) (2026)

## Abstract (arXiv)

Driver activity classification is crucial for ensuring road safety, with applications ranging from driver assistance systems to autonomous vehicle control transitions. In this paper, we present a novel approach leveraging generalizable representations from vision-language models for driver activity classification. Our method employs a Semantic Representation Late Fusion Neural Network (SRLF-Net) to process synchronized video frames from multiple perspectives. Each frame is encoded using a pretrained vision-language encoder, and the resulting embeddings are fused to generate class probability predictions. By leveraging contrastively-learned vision-language representations, our approach achieves robust performance across diverse driver activities. We evaluate our method on the Naturalistic Driving Action Recognition Dataset, demonstrating strong accuracy across many classes. Our results suggest that vision-language representations offer a promising avenue for driver monitoring systems, providing both accuracy and interpretability through natural language descriptors.