# Hazard or Anomaly? Evaluating VLMs for Understanding Dangers and Discrepancies

arXiv: https://arxiv.org/abs/2607.18325 · HTML: https://mlsanity.com/paper/2607.18325/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 18 Jul 2026.

## Builds on (most cited first, 18 of 18)

- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Vision language models are blind: Failing to translate detailed visual features into words](https://mlsanity.com/paper/2407.06581/) (2024)
- [VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding](https://mlsanity.com/paper/2105.09996/) (2021)
- [Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas](https://mlsanity.com/paper/2503.01773/) (2025)
- [Semantic Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2305.11307/) (2023)
- [Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection](https://mlsanity.com/paper/2404.09654/) (2024)
- [VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understanding](https://mlsanity.com/paper/2507.09815/) (2025)
- [ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models](https://mlsanity.com/paper/2401.13311/) (2024)
- [SceneGPT: A Language Model for 3D Scene Understanding](https://mlsanity.com/paper/2408.06926/) (2024)
- [MSTS: A Multimodal Safety Test Suite for Vision-Language Models](https://mlsanity.com/paper/2501.10057/) (2025)
- [Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation](https://mlsanity.com/paper/2310.14025/) (2023)
- [Common Inpainted Objects In-N-Out of Context](https://mlsanity.com/paper/2506.00721/) (2025)
- [Better Safe Than Sorry? Overreaction Problem of Vision Language Models in Visual Emergency Recognition](https://mlsanity.com/paper/2505.15367/) (2025)
- [A VLM-based Method for Visual Anomaly Detection in Robotic Scientific Laboratories](https://mlsanity.com/paper/2506.05405/) (2025)
- [Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalities](https://mlsanity.com/paper/2410.01690/) (2024)
- [Caption This, Reason That: VLMs Caught in the Middle](https://mlsanity.com/paper/2505.21538/) (2025)
- [Real-time Robotics Situation Awareness for Accident Prevention in Industry](https://mlsanity.com/paper/2409.15305/) (2024)

## Abstract (arXiv)

Modern safety-critical systems increasingly rely on human-robot interaction to reduce disaster risk and support decision-making during emergencies. Vision-Language Models (VLMs) are promising for these settings because they can interpret complex scenes and communicate safety-relevant information, but they still require careful evaluation to ensure reliable safety reasoning. In particular, current evaluations often frame danger recognition as a binary decision (Safe/Unsafe), making it unclear whether a model is identifying true physical hazards or merely reacting to unusual scene elements. We address this limitation by introducing an explicit distinction between hazard and anomaly, and by separately recognizing hazardous and anomalous states. We evaluate several state-of-the-art VLMs across two datasets and multiple prompting strategies to test whether this distinction changes model behavior. Our results show that VLMs frequently misinterpret anomalousness as hazardousness, revealing an over-reliance on contextual irregularity as a proxy for danger. We further show that explicitly separating anomaly from hazard provides a more informative evaluation of VLM safety reasoning and exposes failure modes that binary safety judgments can obscure. Our public dataset is available on Roboflow https://app.roboflow.com/vlm-in-context-anomaly-and-hazard-detection/camera-ready-roman-ds.