# Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model

arXiv: https://arxiv.org/abs/2406.15279 · HTML: https://mlsanity.com/paper/2406.15279/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)
- [MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance](https://mlsanity.com/paper/2401.02906/) (2024)
- [How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs](https://mlsanity.com/paper/2311.16101/) (2023)
- [DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback](https://mlsanity.com/paper/2311.10081/) (2023)
- [InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance](https://mlsanity.com/paper/2401.11206/) (2024)
- [Safety of Multimodal Large Language Models on Images and Texts](https://mlsanity.com/paper/2402.00357/) (2024)
- [GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abuse](https://mlsanity.com/paper/2401.01523/) (2024)
- [Assessment of Multimodal Large Language Models in Alignment with Human Values](https://mlsanity.com/paper/2403.17830/) (2024)
- [Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?](https://mlsanity.com/paper/2404.03411/) (2024)
- [Cross-Modal Safety Alignment: Is textual unlearning all you need?](https://mlsanity.com/paper/2406.02575/) (2024)
- [Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks](https://mlsanity.com/paper/2405.04403/) (2024)
- [Large Language Models as Automated Aligners for benchmarking Vision-Language Models](https://mlsanity.com/paper/2311.14580/) (2023)
- [Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models](https://mlsanity.com/paper/2003.11743/) (2020)

## Built on by (most cited first, 26 of 26)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations](https://mlsanity.com/paper/2502.14881/) (2025)
- [SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization](https://mlsanity.com/paper/2511.12982/) (2025)
- [VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization](https://mlsanity.com/paper/2504.12661/) (2025)
- [SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models](https://mlsanity.com/paper/2510.06871/) (2025)
- [How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?](https://mlsanity.com/paper/2410.07571/) (2024)
- [VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety](https://mlsanity.com/paper/2510.18214/) (2025)
- [ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs](https://mlsanity.com/paper/2505.14035/) (2025)
- [Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack](https://mlsanity.com/paper/2505.21967/) (2025)
- [Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory](https://mlsanity.com/paper/2603.15800/) (2026)
- [SIA: Enhancing Safety via Intent Awareness for Vision-Language Models](https://mlsanity.com/paper/2507.16856/) (2025)
- [SafeRedir: Prompt Embedding Redirection for Robust Unlearning in Image Generation Models](https://mlsanity.com/paper/2601.08623/) (2026)
- [Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs](https://mlsanity.com/paper/2603.13292/) (2026)
- [Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix](https://mlsanity.com/paper/2507.09990/) (2025)
- [Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models](https://mlsanity.com/paper/2510.25179/) (2025)
- [Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs](https://mlsanity.com/paper/2604.12616/) (2026)
- [Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Models](https://mlsanity.com/paper/2512.07141/) (2025)
- [COSMO-RL: Towards Trustworthy LMRMs via Joint Safety and Stability](https://mlsanity.com/paper/2510.04196/) (2025)
- [From Sparse Decisions to Dense Reasoning: A Multi-attribute Trajectory Paradigm for Multimodal Moderation](https://mlsanity.com/paper/2602.02536/) (2026)
- [OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences](https://mlsanity.com/paper/2603.09706/) (2026)
- [V-DEAL: Diagnosing Video Safety De-Calibration as an Understanding-Refusal Coupling Failure](https://mlsanity.com/paper/2607.21151/) (2026)
- [Transfer Safety Awareness for Cross-Modal Safety Drift in Multimodal Large Language Models](https://mlsanity.com/paper/2609.02082/) (2026)
- [A2Safe: Counterfactual Evidence-Aligned Adaptive Agent Collaboration for Safe and Effective Visual Question Answering](https://mlsanity.com/paper/2609.24098/) (2026)
- [Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignment](https://mlsanity.com/paper/2609.37837/) (2026)
- [MLLMs Fail to Refuse when Using Tools Agentically](https://mlsanity.com/paper/2610.03938/) (2026)
- [From Local Evidence to Safety Verdicts: Causal Tracing in Vision-Language Models](https://mlsanity.com/paper/2610.07514/) (2026)

## Abstract (arXiv)

As Artificial General Intelligence (AGI) becomes increasingly integrated into various facets of human life, ensuring the safety and ethical alignment of such systems is paramount. Previous studies primarily focus on single-modality threats, which may not suffice given the integrated and complex nature of cross-modality interactions. We introduce a novel safety alignment challenge called Safe Inputs but Unsafe Output (SIUO) to evaluate cross-modality safety alignment. Specifically, it considers cases where single modalities are safe independently but could potentially lead to unsafe or unethical outputs when combined. To empirically investigate this problem, we developed the SIUO, a cross-modality benchmark encompassing 9 critical safety domains, such as self-harm, illegal activities, and privacy violations. Our findings reveal substantial safety vulnerabilities in both closed- and open-source LVLMs, such as GPT-4V and LLaVA, underscoring the inadequacy of current models to reliably interpret and respond to complex, real-world scenarios.