mlsanityData to 7 Oct 2026

ErgoChat: a Visual Query System for the Ergonomic Risk Assessment of Construction Workers

arXiv 2412.19954 ↗First posted 27 Dec 2024cs.CV cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In the construction sector, workers often endure prolonged periods of high-intensity physical work and prolonged use of tools, resulting in injuries and illnesses primarily linked to postural ergonomic risks, a longstanding predominant health concern. To mitigate these risks, researchers have applied various technological methods to identify the ergonomic risks that construction workers face. However, traditional ergonomic risk assessment (ERA) techniques do not offer interactive feedback. The rapidly developing vision-language models (VLMs), capable of generating textual descriptions or answering questions about ergonomic risks based on image inputs, have not yet received widespread attention. This research introduces an interactive visual query system tailored to assess the postural ergonomic risks of construction workers. The system's capabilities include visual question answering (VQA), which responds to visual queries regarding workers' exposure to postural ergonomic risks, and image captioning (IC), which generates textual descriptions of these risks from images. Additionally, this study proposes a dataset designed for training and testing such methodologies. Systematic testing indicates that the VQA functionality delivers an accuracy of 96.5%. Moreover, evaluations using nine metrics for IC and assessments from human experts indicate that the proposed approach surpasses the performance of a method using the same architecture trained solely on generic datasets. This study sets a new direction for future developments in interactive ERA using generative artificial intelligence (AI) technologies.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 16 of its 80 references that are in the map. First posted on arXiv 27 Dec 2024.

Builds on

16 of 80 references.

References · 16

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2016Modeling Context in Referring Expressionscited by 1,054
  8. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  9. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
4 more
  1. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  2. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  3. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  4. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Using Vision Language Models for Safety Hazard Identification in Constructioncited by 12
  2. 2025Integrating Large Language Models for Automated Structural Analysiscited by 12
  3. 2025Automating Structural Engineering Workflows with Large Language Model Agentscited by 6
  4. 2025Fine-Tuning Large Language Models and Evaluating Retrieval Methods for Improved Question Answering on Building Codescited by 2
  5. 2026Vision-Language Models for Ergonomic Assessment of Manual Lifting Tasks: Estimating Horizontal and Vertical Hand Distances from RGB Videocited by 0
  6. 2026Integration of Object Detection and Small VLMs for Construction Safety Hazard Identificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Can Vision-Language Models Understand Construction Workers? An Exploratory Study
  2. 2025Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?
  3. 2025Automating construction safety inspections using a multi-modal vision-language RAG framework
  4. 2025Prompt to Protection: A Comparative Study of Multimodal LLMs in Construction Hazard Recognition
  5. 2024Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safety
  6. 2026Vision-Language Models for Occupational Physical Exposure Assessment: Estimating External Hand Forces in Manual Material Handling Tasks from RGB Video
  7. 2025Automated Hazard Detection in Construction Sites Using Large Language and Vision-Language Models
  8. 2026TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios