# Prompt to Protection: A Comparative Study of Multimodal LLMs in Construction Hazard Recognition

arXiv: https://arxiv.org/abs/2506.07436 · HTML: https://mlsanity.com/paper/2506.07436/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 90 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://mlsanity.com/paper/2212.10001/) (2022)
- [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023)
- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?](https://mlsanity.com/paper/2305.01555/) (2023)
- [Investigating the use of ChatGPT for the scheduling of construction projects](https://mlsanity.com/paper/2302.02805/) (2023)
- [BIM-GPT: a Prompt-Based Virtual Assistant Framework for BIM Information Retrieval](https://mlsanity.com/paper/2304.09333/) (2023)
- [Large Language Models For Text Classification: Case Study And Comprehensive Review](https://mlsanity.com/paper/2501.08457/) (2025)
- [Robot-Enabled Construction Assembly with Automated Sequence Planning based on ChatGPT: RoboGPT](https://mlsanity.com/paper/2304.11018/) (2023)
- [Responsible AI in Construction Safety: Systematic Evaluation of Large Language Models and Prompt Engineering](https://mlsanity.com/paper/2411.08320/) (2024)
- [Multi-Modal Trip Hazard Affordance Detection On Construction Sites](https://mlsanity.com/paper/1706.06718/) (2017)

## Built on by (most cited first, 3 of 3)

- [MonitorVLM:A Vision Language Framework for Safety Violation Detection in Mining Operations](https://mlsanity.com/paper/2510.03666/) (2025)
- [Generative AI for Visualizing Highway Construction Hazards Through Synthetic Images and Temporal Sequences](https://mlsanity.com/paper/2605.11276/) (2026)
- [FearCaut-Qwen: Affective Steering in a Vision-Language Model Shifts the Decision Criterion for Hazard Assessment](https://mlsanity.com/paper/2610.11986/) (2026)

## Abstract (arXiv)

The recent emergence of multimodal large language models (LLMs) has introduced new opportunities for improving visual hazard recognition on construction sites. Unlike traditional computer vision models that rely on domain-specific training and extensive datasets, modern LLMs can interpret and describe complex visual scenes using simple natural language prompts. However, despite growing interest in their applications, there has been limited investigation into how different LLMs perform in safety-critical visual tasks within the construction domain. To address this gap, this study conducts a comparative evaluation of five state-of-the-art LLMs: Claude-3 Opus, GPT-4.5, GPT-4o, GPT-o3, and Gemini 2.0 Pro, to assess their ability to identify potential hazards from real-world construction images. Each model was tested under three prompting strategies: zero-shot, few-shot, and chain-of-thought (CoT). Zero-shot prompting involved minimal instruction, few-shot incorporated basic safety context and a hazard source mnemonic, and CoT provided step-by-step reasoning examples to scaffold model thinking. Quantitative analysis was performed using precision, recall, and F1-score metrics across all conditions. Results reveal that prompting strategy significantly influenced performance, with CoT prompting consistently producing higher accuracy across models. Additionally, LLM performance varied under different conditions, with GPT-4.5 and GPT-o3 outperforming others in most settings. The findings also demonstrate the critical role of prompt design in enhancing the accuracy and consistency of multimodal LLMs for construction safety applications. This study offers actionable insights into the integration of prompt engineering and LLMs for practical hazard recognition, contributing to the development of more reliable AI-assisted safety systems.