Passive Construction Site Safety Monitoring via Persona-Scaffolded Adversarial Chain-of-Thought VLM Verification
Not on the map. Why?
Abstract (arXiv)
Construction remains the deadliest industry sector in the United States, with 1,055 fatal worker injuries recorded in 2023, and the majority preventable. Existing monitoring approaches are expensive, require real-time human operators, or address only a narrow subset of violations. This paper presents a passive, end-of-shift construction safety monitoring pipeline processing video from POV body-worn and fixed wall-mounted cameras through a three-stage architecture: (1) fine-tuned YOLO11 for primary PPE and hazard detection, (2) SAM 3 for segmentation refinement and worker deduplication, and (3) Qwen3-VL-8B-Instruct with a method-prompted, persona-scaffolded three-pass adversarial chain-of-thought protocol for compliance verification and hallucination control. The principal contribution is the Stage 3 prompt design: professional persona backstories following the method-actor framing drive an observed 12% precision improvement over single-pass prompting in an informal three-author review of the 12-video Ironsite development corpus, with the largest gains on hallucination-prone violation categories. Structural message isolation enforces observational independence between a generator, discriminator, and reconciliation pass governed by asymmetric rules encoding priors about human observation versus automated detection reliability. The system maps violations to OSHA standards, performs REBA-inspired ergonomic risk scoring from pose keypoints, and produces per-worker safety reports with timestamped evidence. An evaluation harness is released for future reproduction.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 26 references that are in the map. First posted on arXiv 19 May 2026.
Builds on
5 of 26 references.
References · 5
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2022BoT-SORT: Robust Associations Multi-Pedestrian Trackingcited by 320
- 2026The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Modelscited by 5
- 2024LLMs as Method Actors: A Model for Prompt Engineering and Architecturecited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?
- 2025Using Vision Language Models for Safety Hazard Identification in Construction
- 2026Integration of Object Detection and Small VLMs for Construction Safety Hazard Identification
- 2026SafeBuild-Bench: A Temporal-Robust Construction Safety Benchmark with Graph-Enhanced Data Mining
- 2025Autonomous Construction-Site Safety Inspection Using Mobile Robots: A Multilayer VLM-LLM Pipeline ↗
- 2025Automated Hazard Detection in Construction Sites Using Large Language and Vision-Language Models
- 2025Prompt to Protection: A Comparative Study of Multimodal LLMs in Construction Hazard Recognition
- 2026Can Vision-Language Models Understand Construction Workers? An Exploratory Study