The Use of Multimodal Large Language Models to Detect Objects from Thermal Images: Transportation Applications
On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection
Abstract (arXiv)
The integration of thermal imaging data with Multimodal Large Language Models (MLLMs) constitutes an exciting opportunity for improving the safety and functionality of autonomous driving systems and many Intelligent Transportation Systems (ITS) applications. This study investigates whether MLLMs can understand complex images from RGB and thermal cameras and detect objects directly. Our goals were to 1) assess the ability of the MLLM to learn from information from various sets, 2) detect objects and identify elements in thermal cameras, 3) determine whether two independent modality images show the same scene, and 4) learn all objects using different modalities. The findings showed that both GPT-4 and Gemini were effective in detecting and classifying objects in thermal images. Similarly, the Mean Absolute Percentage Error (MAPE) for pedestrian classification was 70.39% and 81.48%, respectively. Moreover, the MAPE for bike, car, and motorcycle detection were 78.4%, 55.81%, and 96.15%, respectively. Gemini produced MAPE of 66.53%, 59.35% and 78.18% respectively. This finding further demonstrates that MLLM can identify thermal images and can be employed in advanced imaging automation technologies for ITS applications.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 37 references that are in the map. First posted on arXiv 20 Jun 2024.
Builds on
13 of 37 references.
References · 13
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
- 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
- 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 93
- 2023Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Reviewcited by 83
- 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
- 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 58
- 2021GEM: A General Evaluation Benchmark for Multimodal Taskscited by 16
- 2022Overcoming Language Disparity in Online Content Classification with Multimodal Learningcited by 8
- 2022Drone Object Detection Using RGB/IR Fusioncited by 5
Built on by
Citing papers per year: 2024: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing
- 2026ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery
- 2025Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends
- 2024MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarios
- 2025Benchmarking Large and Small MLLMs
- 2026D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models
- 2026Enhancing Multimodal Large Language Models for Safety-Critical Driving Video Analysis
- 2025Zero-Shot Scene Understanding with Multimodal Large Language Models for Automated Vehicles