mlsanityData to 14 Oct 2026

Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?

arXiv 2508.11011 ↗First posted 14 Aug 2025cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Construction safety inspections typically involve a human inspector identifying safety concerns on-site. With the rise of powerful Vision Language Models (VLMs), researchers are exploring their use for tasks such as detecting safety rule violations from on-site images. However, there is a lack of open datasets to comprehensively evaluate and further fine-tune VLMs in construction safety inspection. Current applications of VLMs use small, supervised datasets, limiting their applicability in tasks they are not directly trained for. In this paper, we propose the ConstructionSite 10k, featuring 10,000 construction site images with annotations for three inter-connected tasks, including image captioning, safety rule violation visual question answering (VQA), and construction element visual grounding. Our subsequent evaluation of current state-of-the-art large pre-trained VLMs shows notable generalization abilities in zero-shot and few-shot settings, while additional training is needed to make them applicable to actual construction sites. This dataset allows researchers to train and evaluate their own VLMs with new architectures and techniques, providing a valuable benchmark for construction safety inspection.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 57 references that are in the map. First posted on arXiv 14 Aug 2025.

Builds on

27 of 57 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
15 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  9. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  10. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  12. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  13. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  14. 2023Text encoders bottleneck compositionality in contrastive vision-language modelscited by 34
  15. 2022SODA: Site Object Detection dAtaset for Deep Learning in Constructioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawingscited by 3
  2. 2026Integration of Object Detection and Small VLMs for Construction Safety Hazard Identificationcited by 0
  3. 2026General Hazard Detectioncited by 0
  4. 2026AVA-VLM: Adaptive Visual Attention-Vision Language Model for In-the-Wild Construction Site Monitoringcited by 0
  5. 2026Visual Compliance via Executable Safety Rule Entailmentcited by 0
  6. 2026A paired synthetic construction-site image dataset for robust computer vision under adverse conditionscited by 0
  7. 2026RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Workcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Using Vision Language Models for Safety Hazard Identification in Construction
  2. 2025Automated Hazard Detection in Construction Sites Using Large Language and Vision-Language Models
  3. 2025Automating construction safety inspections using a multi-modal vision-language RAG framework
  4. 2024Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safety
  5. 2025Prompt to Protection: A Comparative Study of Multimodal LLMs in Construction Hazard Recognition
  6. 2026Can Vision-Language Models Understand Construction Workers? An Exploratory Study
  7. 2024Urban Safety Perception Assessments via Integrating Multimodal Large Language Models with Street View Images
  8. 2024Evaluation and Comparison of Visual Language Models for Transportation Engineering Problems