# When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily Life

arXiv: https://arxiv.org/abs/2601.04043 · HTML: https://mlsanity.com/paper/2601.04043/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)
- [Imagine while Reasoning in Space: Multimodal Visualization-of-Thought](https://mlsanity.com/paper/2501.07542/) (2025)
- [Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation](https://mlsanity.com/paper/2403.09572/) (2024)
- [DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback](https://mlsanity.com/paper/2311.10081/) (2023)
- [VLSBench: Unveiling Visual Leakage in Multimodal Safety](https://mlsanity.com/paper/2411.19939/) (2024)
- [MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?](https://mlsanity.com/paper/2406.17806/) (2024)
- [Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs](https://mlsanity.com/paper/2502.11184/) (2025)
- [Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Model](https://mlsanity.com/paper/2505.06538/) (2025)
- [Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks](https://mlsanity.com/paper/2405.04403/) (2024)
- [OmniGeo: Towards a Multimodal Large Language Models for Geospatial Artificial Intelligence](https://mlsanity.com/paper/2503.16326/) (2025)
- [Towards Evaluating Proactive Risk Awareness of Multimodal Language Models](https://mlsanity.com/paper/2505.17455/) (2025)
- [Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures](https://mlsanity.com/paper/2506.07402/) (2025)

## Built on by (most cited first, 3 of 3)

- [Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language Models](https://mlsanity.com/paper/2604.01444/) (2026)
- [TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios](https://mlsanity.com/paper/2603.29759/) (2026)
- [SAVOR: Self-Aware Visual Grounding via Confidence-Calibrated Reinforcement Learning for Multimodal Hallucination Mitigation](https://mlsanity.com/paper/2609.16601/) (2026)

## Abstract (arXiv)

As Multimodal Large Language Models (MLLMs) become an indispensable assistant in human life, the unsafe content generated by MLLMs poses a danger to human behavior, perpetually overhanging human society like a sword of Damocles. To investigate and evaluate the safety impact of MLLMs responses on human behavior in daily life, we introduce SaLAD, a multimodal safety benchmark which contains 2,013 real-world image-text samples across 10 common categories, with a balanced design covering both unsafe scenarios and cases of oversensitivity. It emphasizes realistic risk exposure, authentic visual inputs, and fine-grained cross-modal reasoning, ensuring that safety risks cannot be inferred from text alone. We further propose a safety-warning-based evaluation framework that encourages models to provide clear and informative safety warnings, rather than generic refusals. Results on 18 MLLMs demonstrate that the top-performing models achieve a safe response rate of only 57.2% on unsafe queries. Moreover, even popular safety alignment methods limit effectiveness of the models in our scenario, revealing the vulnerabilities of current MLLMs in identifying dangerous behaviors in daily life. Our dataset is available at https://github.com/xinyuelou/SaLAD.