# Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing

arXiv: https://arxiv.org/abs/2409.18286 · HTML: https://mlsanity.com/paper/2409.18286/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 85 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 28 of 28)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving](https://mlsanity.com/paper/2310.03026/) (2023)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [Contextual Object Detection with Multimodal Large Language Models](https://mlsanity.com/paper/2305.18279/) (2023)
- [Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles](https://mlsanity.com/paper/2310.08034/) (2023)
- [Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models](https://mlsanity.com/paper/2401.00625/) (2024)
- [Multi-Modal 3D Object Detection in Autonomous Driving: a Survey](https://mlsanity.com/paper/2106.12735/) (2021)
- [RRPN: Radar Region Proposal Network for Object Detection in Autonomous Vehicles](https://mlsanity.com/paper/1905.00526/) (2019)
- [GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models](https://mlsanity.com/paper/2312.03543/) (2023)
- [Exploring Perceptual Limitation of Multimodal Large Language Models](https://mlsanity.com/paper/2402.07384/) (2024)
- [Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives](https://mlsanity.com/paper/2402.02968/) (2024)
- [Visual Reasoning and Multi-Agent Approach in Multimodal Large Language Models (MLLMs): Solving TSP and mTSP Combinatorial Challenges](https://mlsanity.com/paper/2407.00092/) (2024)
- [Law of Vision Representation in MLLMs](https://mlsanity.com/paper/2408.16357/) (2024)
- [Deep Learning Serves Traffic Safety Analysis: A Forward-looking Review](https://mlsanity.com/paper/2203.10939/) (2022)
- [Generative AI for Enhancing Active Learning in Education: A Comparative Study of GPT-3.5 and GPT-4 in Crafting Customized Test Questions](https://mlsanity.com/paper/2406.13903/) (2024)
- [Exploring Traffic Crash Narratives in Jordan Using Text Mining Analytics](https://mlsanity.com/paper/2406.09438/) (2024)
- [Object Detection using Oriented Window Learning Vi-sion Transformer: Roadway Assets Recognition](https://mlsanity.com/paper/2406.10712/) (2024)
- [Leveraging Large Language Models (LLMs) for Traffic Management at Urban Intersections: The Case of Mixed Traffic Scenarios](https://mlsanity.com/paper/2408.00948/) (2024)
- [Question-Answering (QA) Model for a Personalized Learning Assistant for Arabic Language](https://mlsanity.com/paper/2406.08519/) (2024)
- [Transformer Models in Education: Summarizing Science Textbooks with AraBART, MT5, AraT5, and mBART](https://mlsanity.com/paper/2406.07692/) (2024)
- [Automated Question Generation for Science Tests in Arabic Language Using NLP Techniques](https://mlsanity.com/paper/2406.08520/) (2024)
- [The Devil Is in the Details: An Efficient Convolutional Neural Network for Transport Mode Detection](https://mlsanity.com/paper/2109.09504/) (2021)
- [Automated Pavement Cracks Detection and Classification Using Deep Learning](https://mlsanity.com/paper/2406.07674/) (2024)
- [Advancing Roadway Sign Detection with YOLO Models and Transfer Learning](https://mlsanity.com/paper/2406.09437/) (2024)

## Built on by (most cited first, 10 of 10)

- [Object Detection with Multimodal Large Vision-Language Models: An In-depth Review](https://mlsanity.com/paper/2508.19294/) (2025)
- [Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigm](https://mlsanity.com/paper/2411.10869/) (2024)
- [Transit Pulse: Utilizing Social Media as a Source for Customer Feedback and Information Extraction with Large Language Model](https://mlsanity.com/paper/2410.15016/) (2024)
- [HazardNet: A Small-Scale Vision Language Model for Real-Time Traffic Safety Detection at Edge Devices](https://mlsanity.com/paper/2502.20572/) (2025)
- [Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends](https://mlsanity.com/paper/2504.16134/) (2025)
- [Visual Reasoning at Urban Intersections: FineTuning GPT-4o for Traffic Conflict Detection](https://mlsanity.com/paper/2502.20573/) (2025)
- [Network Traffic Classification Using Machine Learning, Transformer, and Large Language Models](https://mlsanity.com/paper/2503.02141/) (2025)
- [Zero-Shot Scene Understanding with Multimodal Large Language Models for Automated Vehicles](https://mlsanity.com/paper/2506.12232/) (2025)
- [Advanced Assistance for Traffic Crash Analysis: An AI-Driven Multi-Agent Approach to Pre-Crash Reconstruction](https://mlsanity.com/paper/2511.10853/) (2025)
- [Membership Inference for Contrastive Pre-training Models with Text-only PII Queries](https://mlsanity.com/paper/2603.14222/) (2026)

## Abstract (arXiv)

This study aims to comprehensively review and empirically evaluate the application of multimodal large language models (MLLMs) and Large Vision Models (VLMs) in object detection for transportation systems. In the first fold, we provide a background about the potential benefits of MLLMs in transportation applications and conduct a comprehensive review of current MLLM technologies in previous studies. We highlight their effectiveness and limitations in object detection within various transportation scenarios. The second fold involves providing an overview of the taxonomy of end-to-end object detection in transportation applications and future directions. Building on this, we proposed empirical analysis for testing MLLMs on three real-world transportation problems that include object detection tasks namely, road safety attributes extraction, safety-critical event detection, and visual reasoning of thermal images. Our findings provide a detailed assessment of MLLM performance, uncovering both strengths and areas for improvement. Finally, we discuss practical limitations and challenges of MLLMs in enhancing object detection in transportation, thereby offering a roadmap for future research and development in this critical area.