# Evaluating Vision-Language Models for Zero-Shot Detection, Classification, and Association of Motorcycles, Passengers, and Helmets

arXiv: https://arxiv.org/abs/2408.02244 · HTML: https://mlsanity.com/paper/2408.02244/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Scaling Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.09683/) (2023)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [NMS Strikes Back](https://mlsanity.com/paper/2212.06137/) (2022)
- [Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets](https://mlsanity.com/paper/2402.07320/) (2024)
- [The 7th AI City Challenge](https://mlsanity.com/paper/2304.07500/) (2023)
- [The Why, When, and How to Use Active Learning in Large-Data-Driven 3D Object Detection for Safe Autonomous Driving: An Empirical Exploration](https://mlsanity.com/paper/2401.16634/) (2024)
- [ActiveAnno3D -- An Active Learning Framework for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2402.03235/) (2024)
- [The 8th AI City Challenge](https://mlsanity.com/paper/2404.09432/) (2024)
- [Salient Sign Detection In Safe Autonomous Driving: AI Which Reasons Over Full Visual Context](https://mlsanity.com/paper/2301.05804/) (2023)
- [On Salience-Sensitive Sign Classification in Autonomous Vehicle Path Planning: Experimental Explorations with a Novel Dataset](https://mlsanity.com/paper/2112.00942/) (2021)
- [Pedestrian Behavior Maps for Safety Advisories: CHAMP Framework and Real-World Data Analysis](https://mlsanity.com/paper/2305.04506/) (2023)
- [Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognition](https://mlsanity.com/paper/2301.12592/) (2023)

## Built on by (most cited first, 4 of 4)

- [Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safety](https://mlsanity.com/paper/2410.12225/) (2024)
- [Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning](https://mlsanity.com/paper/2602.07680/) (2026)
- [Beyond General Prompts: Automated Prompt Refinement using Contrastive Class Alignment Scores for Disambiguating Objects in Vision-Language Models](https://mlsanity.com/paper/2505.09139/) (2025)
- [Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making](https://mlsanity.com/paper/2602.07668/) (2026)

## Abstract (arXiv)

Motorcycle accidents pose significant risks, particularly when riders and passengers do not wear helmets. This study evaluates the efficacy of an advanced vision-language foundation model, OWLv2, in detecting and classifying various helmet-wearing statuses of motorcycle occupants using video data. We extend the dataset provided by the CVPR AI City Challenge and employ a cascaded model approach for detection and classification tasks, integrating OWLv2 and CNN models. The results highlight the potential of zero-shot learning to address challenges arising from incomplete and biased training datasets, demonstrating the usage of such models in detecting motorcycles, helmet usage, and occupant positions under varied conditions. We have achieved an average precision of 0.5324 for helmet detection and provided precision-recall curves detailing the detection and classification performance. Despite limitations such as low-resolution data and poor visibility, our research shows promising advancements in automated vehicle safety and traffic safety enforcement systems.