mlsanityData to 7 Oct 2026

Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing

arXiv 2409.18286 ↗First posted 26 Sep 2024cs.CV cs.AI cs.CLcited by 10

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

This study aims to comprehensively review and empirically evaluate the application of multimodal large language models (MLLMs) and Large Vision Models (VLMs) in object detection for transportation systems. In the first fold, we provide a background about the potential benefits of MLLMs in transportation applications and conduct a comprehensive review of current MLLM technologies in previous studies. We highlight their effectiveness and limitations in object detection within various transportation scenarios. The second fold involves providing an overview of the taxonomy of end-to-end object detection in transportation applications and future directions. Building on this, we proposed empirical analysis for testing MLLMs on three real-world transportation problems that include object detection tasks namely, road safety attributes extraction, safety-critical event detection, and visual reasoning of thermal images. Our findings provide a detailed assessment of MLLM performance, uncovering both strengths and areas for improvement. Finally, we discuss practical limitations and challenges of MLLMs in enhancing object detection in transportation, thereby offering a roadmap for future research and development in this critical area.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 85 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

28 of 85 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 27

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  4. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  5. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  6. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  7. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  8. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  9. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 64
  10. 2021Multi-Modal 3D Object Detection in Autonomous Driving: a Surveycited by 63
  11. 2019RRPN: Radar Region Proposal Network for Object Detection in Autonomous Vehiclescited by 33
  12. 2023GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Modelscited by 31
15 more
  1. 2024Exploring Perceptual Limitation of Multimodal Large Language Modelscited by 24
  2. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  3. 2024Visual Reasoning and Multi-Agent Approach in Multimodal Large Language Models (MLLMs): Solving TSP and mTSP Combinatorial Challengescited by 21
  4. 2024Law of Vision Representation in MLLMscited by 16
  5. 2022Deep Learning Serves Traffic Safety Analysis: A Forward-looking Reviewcited by 13
  6. 2024Generative AI for Enhancing Active Learning in Education: A Comparative Study of GPT-3.5 and GPT-4 in Crafting Customized Test Questionscited by 8
  7. 2024Exploring Traffic Crash Narratives in Jordan Using Text Mining Analyticscited by 6
  8. 2024Object Detection using Oriented Window Learning Vi-sion Transformer: Roadway Assets Recognitioncited by 6
  9. 2024Leveraging Large Language Models (LLMs) for Traffic Management at Urban Intersections: The Case of Mixed Traffic Scenarioscited by 6
  10. 2024Question-Answering (QA) Model for a Personalized Learning Assistant for Arabic Languagecited by 5
  11. 2024Transformer Models in Education: Summarizing Science Textbooks with AraBART, MT5, AraT5, and mBARTcited by 3
  12. 2024Automated Question Generation for Science Tests in Arabic Language Using NLP Techniquescited by 3
  13. 2021The Devil Is in the Details: An Efficient Convolutional Neural Network for Transport Mode Detectioncited by 2
  14. 2024Automated Pavement Cracks Detection and Classification Using Deep Learningcited by 2
  15. 2024Advancing Roadway Sign Detection with YOLO Models and Transfer Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 1.

10 papers.

  1. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  2. 2024Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigmcited by 12
  3. 2024Transit Pulse: Utilizing Social Media as a Source for Customer Feedback and Information Extraction with Large Language Modelcited by 1
  4. 2025HazardNet: A Small-Scale Vision Language Model for Real-Time Traffic Safety Detection at Edge Devicescited by 1
  5. 2025Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trendscited by 1
  6. 2025Visual Reasoning at Urban Intersections: FineTuning GPT-4o for Traffic Conflict Detectioncited by 0
  7. 2025Network Traffic Classification Using Machine Learning, Transformer, and Large Language Modelscited by 0
  8. 2025Zero-Shot Scene Understanding with Multimodal Large Language Models for Automated Vehiclescited by 0
  9. 2025Advanced Assistance for Traffic Crash Analysis: An AI-Driven Multi-Agent Approach to Pre-Crash Reconstructioncited by 0
  10. 2026Membership Inference for Contrastive Pre-training Models with Text-only PII Queriescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Use of Multimodal Large Language Models to Detect Objects from Thermal Images: Transportation Applications
  2. 2025Large Language Models for Crash Detection in Video: A Survey of Methods, Datasets, and Challenges
  3. 2025Investigating Traffic Accident Detection Using Multimodal Large Language Models
  4. 2024MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarios
  5. 2026A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions
  6. 2025All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles
  7. 2025Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
  8. 2024Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events