# Using Vision Language Models for Safety Hazard Identification in Construction

arXiv: https://arxiv.org/abs/2504.09083 · HTML: https://mlsanity.com/paper/2504.09083/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 64 references that are in the map.
- First posted on arXiv 12 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [Multi-Object Hallucination in Vision-Language Models](https://mlsanity.com/paper/2407.06192/) (2024)
- [Unveiling Security, Privacy, and Ethical Concerns of ChatGPT](https://mlsanity.com/paper/2307.14192/) (2023)
- [GPT Models in Construction Industry: Opportunities, Limitations, and a Use Case Validation](https://mlsanity.com/paper/2305.18997/) (2023)
- [ErgoChat: a Visual Query System for the Ergonomic Risk Assessment of Construction Workers](https://mlsanity.com/paper/2412.19954/) (2024)
- [Autonomous Prompt Engineering in Large Language Models](https://mlsanity.com/paper/2407.11000/) (2024)
- [From classical techniques to convolution-based models: A review of object detection algorithms](https://mlsanity.com/paper/2412.05252/) (2024)
- [Demystifying the Potential of ChatGPT-4 Vision for Construction Progress Monitoring](https://mlsanity.com/paper/2412.16108/) (2024)

## Built on by (most cited first, 13 of 13)

- [Automating Structural Engineering Workflows with Large Language Model Agents](https://mlsanity.com/paper/2510.11004/) (2025)
- [MonitorVLM:A Vision Language Framework for Safety Violation Detection in Mining Operations](https://mlsanity.com/paper/2510.03666/) (2025)
- [Fine-Tuning Large Language Models and Evaluating Retrieval Methods for Improved Question Answering on Building Codes](https://mlsanity.com/paper/2505.04666/) (2025)
- [Bridging Dual Knowledge Graphs for Multi-Hop Question Answering in Construction Safety](https://mlsanity.com/paper/2507.13625/) (2025)
- [Automated Facility Enumeration for Building Compliance Checking using Door Detection and Large Language Models](https://mlsanity.com/paper/2509.17283/) (2025)
- [Generative AI for Visualizing Highway Construction Hazards Through Synthetic Images and Temporal Sequences](https://mlsanity.com/paper/2605.11276/) (2026)
- [Table Comprehension in Building Codes using Vision Language Models and Domain-Specific Fine-Tuning](https://mlsanity.com/paper/2511.18306/) (2025)
- [Integration of Object Detection and Small VLMs for Construction Safety Hazard Identification](https://mlsanity.com/paper/2604.05210/) (2026)
- [Task-Aligned Stability Analysis of Vision-Language Models for Autonomous Driving Hazard Detection](https://mlsanity.com/paper/2606.11889/) (2026)
- [Multi-modal Rail Crossing Safety Analysis](https://mlsanity.com/paper/2607.01365/) (2026)
- [SafeBuild-Bench: A Temporal-Robust Construction Safety Benchmark with Graph-Enhanced Data Mining](https://mlsanity.com/paper/2608.00068/) (2026)
- [SafeSceneReason: A Multimodal Reasoning Benchmark Connecting Industrial Hazards with Accident Knowledge](https://mlsanity.com/paper/2608.09230/) (2026)
- [Agentic AI with Structured CoT for Enhancing AI's Spatial Intelligence: Visualization and Reasoning of Rotation](https://mlsanity.com/paper/2610.04188/) (2026)

## Abstract (arXiv)

Safety hazard identification and prevention are the key elements of proactive safety management. Previous research has extensively explored the applications of computer vision to automatically identify hazards from image clips collected from construction sites. However, these methods struggle to identify context-specific hazards, as they focus on detecting predefined individual entities without understanding their spatial relationships and interactions. Furthermore, their limited adaptability to varying construction site guidelines and conditions hinders their generalization across different projects. These limitations reduce their ability to assess hazards in complex construction environments and adaptability to unseen risks, leading to potential safety gaps. To address these challenges, we proposed and experimentally validated a Vision Language Model (VLM)-based framework for the identification of construction hazards. The framework incorporates a prompt engineering module that structures safety guidelines into contextual queries, allowing VLM to process visual information and generate hazard assessments aligned with the regulation guide. Within this framework, we evaluated state-of-the-art VLMs, including GPT-4o, Gemini, Llama 3.2, and InternVL2, using a custom dataset of 1100 construction site images. Experimental results show that GPT-4o and Gemini 1.5 Pro outperformed alternatives and displayed promising BERTScore of 0.906 and 0.888 respectively, highlighting their ability to identify both general and context-specific hazards. However, processing times remain a significant challenge, impacting real-time feasibility. These findings offer insights into the practical deployment of VLMs for construction site hazard detection, thereby contributing to the enhancement of proactive safety management.