# Evaluation and Comparison of Visual Language Models for Transportation Engineering Problems

arXiv: https://arxiv.org/abs/2409.02278 · HTML: https://mlsanity.com/paper/2409.02278/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS](https://mlsanity.com/paper/2304.00501/) (2023)
- [Video Anomaly Detection and Explanation via Large Language Models](https://mlsanity.com/paper/2401.05702/) (2024)
- [Vision-Language Models can Identify Distracted Driver Behavior from Naturalistic Videos](https://mlsanity.com/paper/2306.10159/) (2023)
- [The 8th AI City Challenge](https://mlsanity.com/paper/2404.09432/) (2024)

## Built on by (most cited first, 4 of 4)

- [Is this Generated Person Existed in Real-world? Fine-grained Detecting and Calibrating Abnormal Human-body](https://mlsanity.com/paper/2411.14205/) (2024)
- [Transit Pulse: Utilizing Social Media as a Source for Customer Feedback and Information Extraction with Large Language Model](https://mlsanity.com/paper/2410.15016/) (2024)
- [Scene-Aware Vectorized Memory Multi-Agent Framework with Cross-Modal Differentiated Quantization VLMs for Visually Impaired Assistance](https://mlsanity.com/paper/2508.18177/) (2025)
- [Mechanistic Interpretability-Guided Selective Fine-Tuning of Vision-Language Models for Centimeter-Level Flood Depth Estimation](https://mlsanity.com/paper/2608.07562/) (2026)

## Abstract (arXiv)

Recent developments in vision language models (VLM) have shown great potential for diverse applications related to image understanding. In this study, we have explored state-of-the-art VLM models for vision-based transportation engineering tasks such as image classification and object detection. The image classification task involves congestion detection and crack identification, whereas, for object detection, helmet violations were identified. We have applied open-source models such as CLIP, BLIP, OWL-ViT, Llava-Next, and closed-source GPT-4o to evaluate the performance of these state-of-the-art VLM models to harness the capabilities of language understanding for vision-based transportation tasks. These tasks were performed by applying zero-shot prompting to the VLM models, as zero-shot prompting involves performing tasks without any training on those tasks. It eliminates the need for annotated datasets or fine-tuning for specific tasks. Though these models gave comparative results with benchmark Convolutional Neural Networks (CNN) models in the image classification tasks, for object localization tasks, it still needs improvement. Therefore, this study provides a comprehensive evaluation of the state-of-the-art VLM models highlighting the advantages and limitations of the models, which can be taken as the baseline for future improvement and wide-scale implementation.