# Better Safe Than Sorry? Overreaction Problem of Vision Language Models in Visual Emergency Recognition

arXiv: https://arxiv.org/abs/2505.15367 · HTML: https://mlsanity.com/paper/2505.15367/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 18 of 18)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs](https://mlsanity.com/paper/2311.16101/) (2023)
- [Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images](https://mlsanity.com/paper/2402.05779/) (2024)
- [ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understanding](https://mlsanity.com/paper/2410.00982/) (2024)
- [Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles](https://mlsanity.com/paper/2211.16504/) (2022)
- [ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments](https://mlsanity.com/paper/2410.06420/) (2024)
- [Leveraging Large Language Models for Enhanced Digital Twin Modeling: Trends, Methods, and Challenges](https://mlsanity.com/paper/2503.02167/) (2025)
- [Evaluation of Safety Cognition Capability in Vision-Language Models for Autonomous Driving](https://mlsanity.com/paper/2503.06497/) (2025)
- ["What If Smart Homes Could See Our Homes?": Exploring DIY Smart Home Building Experiences with VLM-Based Camera Sensors](https://mlsanity.com/paper/2503.02816/) (2025)
- [Balancing Complexity and Informativeness in LLM-Based Clustering: Finding the Goldilocks Zone](https://mlsanity.com/paper/2504.04314/) (2025)

## Built on by (most cited first, 7 of 7)

- [Jailbreaking on Text-to-Video Models via Scene Splitting Strategy](https://mlsanity.com/paper/2509.22292/) (2025)
- [When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models](https://mlsanity.com/paper/2605.27851/) (2026)
- [EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards](https://mlsanity.com/paper/2607.00218/) (2026)
- [SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safety](https://mlsanity.com/paper/2603.02635/) (2026)
- [Autonomous VR-Based Risk Detection for Situational Awareness in Dangerous Settings](https://mlsanity.com/paper/2607.16582/) (2026)
- [Hazard or Anomaly? Evaluating VLMs for Understanding Dangers and Discrepancies](https://mlsanity.com/paper/2607.18325/) (2026)
- [Visual Compliance via Executable Safety Rule Entailment](https://mlsanity.com/paper/2609.18328/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) have shown capabilities in interpreting visual content, but their reliability in safety-critical scenarios remains insufficiently explored. We introduce VERI, a diagnostic benchmark comprising 200 synthetic images (100 contrastive pairs) and an additional 50 real-world images (25 pairs) for validation. Each emergency scene is paired with a visually similar but safe counterpart through human verification. Using a two-stage evaluation protocol (risk identification and emergency response), we assess 17 VLMs across medical emergencies, accidents, and natural disasters. Our analysis reveals an "overreaction problem": models achieve high recall (70-100%) but suffer from low precision, misclassifying 31-96% of safe situations as dangerous. Seven safe scenarios were universally misclassified by all models. This "better-safe-than-sorry" bias stems from contextual overinterpretation (88-98% of errors). Both synthetic and real-world datasets confirm these systematic patterns, challenging VLM reliability in safety-critical applications. Addressing this requires enhanced contextual reasoning in ambiguous visual situations.