# iSafetyBench: A video-language benchmark for safety in industrial environment

arXiv: https://arxiv.org/abs/2508.00399 · HTML: https://mlsanity.com/paper/2508.00399/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 1 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 24 of 24)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [A Short Note on the Kinetics-700 Human Action Dataset](https://mlsanity.com/paper/1907.06987/) (2019)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization](https://mlsanity.com/paper/2411.10442/) (2024)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling](https://mlsanity.com/paper/2501.00574/) (2024)
- [Ovis: Structural Embedding Alignment for Multimodal Large Language Model](https://mlsanity.com/paper/2405.20797/) (2024)
- [Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution](https://mlsanity.com/paper/2409.12961/) (2024)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [MEVA: A Large-Scale Multiview, Multimodal Video Dataset for Activity Detection](https://mlsanity.com/paper/2012.00914/) (2020)
- [Valley2: Exploring Multimodal Models with Scalable Vision-Language Design](https://mlsanity.com/paper/2501.05901/) (2025)
- [OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments](https://mlsanity.com/paper/2212.11152/) (2022)
- [TIMo -- A Dataset for Indoor Building Monitoring with a Time-of-Flight Camera](https://mlsanity.com/paper/2108.12196/) (2021)

## Built on by (most cited first, 5 of 5)

- [RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classification](https://mlsanity.com/paper/2511.15923/) (2025)
- [SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments](https://mlsanity.com/paper/2607.05264/) (2026)
- [Learning to Deny: Action Denial in Multimodal Large Language Models](https://mlsanity.com/paper/2606.31187/) (2026)
- [SafeSceneReason: A Multimodal Reasoning Benchmark Connecting Industrial Hazards with Accident Knowledge](https://mlsanity.com/paper/2608.09230/) (2026)
- [LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios](https://mlsanity.com/paper/2609.09790/) (2026)

## Abstract (arXiv)

Recent advances in vision-language models (VLMs) have enabled impressive generalization across diverse video understanding tasks under zero-shot settings. However, their capabilities in high-stakes industrial domains-where recognizing both routine operations and safety-critical anomalies is essential-remain largely underexplored. To address this gap, we introduce iSafetyBench, a new video-language benchmark specifically designed to evaluate model performance in industrial environments across both normal and hazardous scenarios. iSafetyBench comprises 1,100 video clips sourced from real-world industrial settings, annotated with open-vocabulary, multi-label action tags spanning 98 routine and 67 hazardous action categories. Each clip is paired with multiple-choice questions for both single-label and multi-label evaluation, enabling fine-grained assessment of VLMs in both standard and safety-critical contexts. We evaluate eight state-of-the-art video-language models under zero-shot conditions. Despite their strong performance on existing video benchmarks, these models struggle with iSafetyBench-particularly in recognizing hazardous activities and in multi-label scenarios. Our results reveal significant performance gaps, underscoring the need for more robust, safety-aware multimodal models for industrial applications. iSafetyBench provides a first-of-its-kind testbed to drive progress in this direction. The dataset is available at: https://github.com/iSafetyBench/data.