Multi2AV-Safety: Benchmarking Safety in Multimodal-to-Audio-Video Generation
Not on the map. Why?
Abstract (arXiv)
Recent audio-video generators increasingly support joint conditioning on text, images, audio, and video. These capabilities also enable attacks that exploit cross-modal interactions or obscure harmful intent to bypass safeguards and induce harmful audio-video outputs. However, existing generation-safety benchmarks have not kept pace with these advances, providing limited coverage of multimodal input combinations and obscured attack intents. To address these gaps, we introduce Multi2AV-Safety, the first full-coverage red-team benchmark for multimodal-to-audio-video generation, comprising 11,024 attack instances across all 11 non-singleton T/I/A/V conditioning configurations, 4 attack-intent categories, and 5 harm categories. Our evaluation of recent state-of-the-art models, including four multimodal-conditioned audio-video generators and eight safety guards, reveals substantial vulnerabilities in both generation and safeguarding, with multimodal compositional risk and obscured attack-intent risk emerging as two complementary challenges. Guided by these findings, we introduce PerceptGuard, an omni-modal guard integrating compositional-risk and attack-intent supervision through structured risk perception learning. By jointly training rationale generation and safety classification, it learns shared risk representations that enable a safety head to make efficient predictions at inference without rationale decoding, while retaining the ability to generate explanations on demand. Across 34 safety benchmarks, PerceptGuard combines SOTA multimodal safety detection with highly competitive unimodal performance, strengthening input-side safeguards against multimodal attacks on omni models. In particular, it improves safeguarding against the above risks, achieving an overall recall of 86.06% on Multi2AV-Safety and outperforming GuardReasoner-Omni by 14.56%.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 41 references that are in the map. First posted on arXiv 27 Aug 2026.
Builds on
35 of 41 references.
References · 35
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
- 2025Qwen3-Omni Technical Reportcited by 451
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
- 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
- 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 239
- 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
- 2023SneakyPrompt: Jailbreaking Text-to-image Generative Modelscited by 175
- 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Promptscited by 161
- 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 118
- 2024Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Modelscited by 68
- 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
23 more
- 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
- 2024Perception-guided Jailbreak against Text-to-Image Modelscited by 37
- 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 37
- 2024SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Datasetcited by 34
- 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 33
- 2026HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformercited by 29
- 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
- 2026Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Modelcited by 23
- 2024Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generationcited by 20
- 2024DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimizationcited by 19
- 2025T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Modelcited by 16
- 2024Multimodal Pragmatic Jailbreak on Text-to-image Modelscited by 13
- 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 13
- 2023Harnessing LLM to Attack LLM-Guarded Text-to-Image Modelscited by 10
- 2024AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion modelscited by 10
- 2025Jailbreaking on Text-to-Video Models via Scene Splitting Strategycited by 10
- 2025ConceptGuard: Proactive Safety in Text-and-Image-to-Video Generation through Multimodal Risk Detectioncited by 5
- 2025GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Supervisioncited by 4
- 2026SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generationcited by 3
- 2025Contextual Image Attack: How Visual Context Exposes Multimodal Safety Vulnerabilitiescited by 2
- 2026GuardReasoner-Omni: A Reasoning-based Multi-modal Guardrail for Text, Image, Video, and Audiocited by 2
- 2026UniSAFE: A Comprehensive Benchmark for Safety Evaluation of Unified Multimodal Modelscited by 1
- 2026VPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attackscited by 1
Built on by
Not cited yet.