# T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models

arXiv: https://arxiv.org/abs/2407.05965 · HTML: https://mlsanity.com/paper/2407.05965/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 68 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 30 of 50)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2311.17600/) (2023)
- [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022)
- [T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation](https://mlsanity.com/paper/2307.06350/) (2023)
- [Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?](https://mlsanity.com/paper/2310.10012/) (2023)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)

## Built on by (most cited first, 30 of 66)

- [SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation](https://mlsanity.com/paper/2410.12761/) (2024)
- [VideoRAG: Retrieval-Augmented Generation over Video Corpus](https://mlsanity.com/paper/2501.05874/) (2025)
- [Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback](https://mlsanity.com/paper/2412.02617/) (2024)
- [SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations](https://mlsanity.com/paper/2412.06878/) (2024)
- [Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey](https://mlsanity.com/paper/2411.09259/) (2024)
- [The Dawn of Video Generation: Preliminary Explorations with SORA-like Models](https://mlsanity.com/paper/2410.05227/) (2024)
- [VPO: Aligning Text-to-Video Generation Models with Prompt Optimization](https://mlsanity.com/paper/2503.20491/) (2025)
- [CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation](https://mlsanity.com/paper/2505.15145/) (2025)
- [T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attacks](https://mlsanity.com/paper/2505.06679/) (2025)
- [DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios](https://mlsanity.com/paper/2510.15501/) (2025)
- [T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Models](https://mlsanity.com/paper/2504.15512/) (2025)
- [T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model](https://mlsanity.com/paper/2510.22300/) (2025)
- [Survey of Adversarial Robustness in Multimodal Large Language Models](https://mlsanity.com/paper/2503.13962/) (2025)
- [Survey of Video Diffusion Models: Foundations, Implementations, and Applications](https://mlsanity.com/paper/2504.16081/) (2025)
- [VideoEraser: Concept Erasure in Text-to-Video Diffusion Models](https://mlsanity.com/paper/2508.15314/) (2025)
- [Video Unlearning via Low-Rank Refusal Vector](https://mlsanity.com/paper/2506.07891/) (2025)
- [Jailbreaking on Text-to-Video Models via Scene Splitting Strategy](https://mlsanity.com/paper/2509.22292/) (2025)
- [When Safety Collides: Resolving Multi-Category Harmful Conflicts in Text-to-Image Diffusion via Adaptive Safety Guidance](https://mlsanity.com/paper/2602.20880/) (2026)
- [Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Models](https://mlsanity.com/paper/2504.04051/) (2025)
- [BadVideo: Stealthy Backdoor Attack against Text-to-Video Generation](https://mlsanity.com/paper/2504.16907/) (2025)
- [Red-Teaming Text-to-Image Systems by Rule-based Preference Modeling](https://mlsanity.com/paper/2505.21074/) (2025)
- [RunawayEvil: Jailbreaking the Image-to-Video Generative Models](https://mlsanity.com/paper/2512.06674/) (2025)
- [Autonomous LLM-Enhanced Adversarial Attack for Text-to-Motion](https://mlsanity.com/paper/2408.00352/) (2024)
- [GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning](https://mlsanity.com/paper/2503.02341/) (2025)
- [VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Models](https://mlsanity.com/paper/2602.20999/) (2026)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [T2VEval: Benchmark Dataset and Objective Evaluation Method for T2V-generated Videos](https://mlsanity.com/paper/2501.08545/) (2025)
- [AEGIS: Authenticity Evaluation Benchmark for AI-Generated Video Sequences](https://mlsanity.com/paper/2508.10771/) (2025)
- [A Granular Study of Safety Pretraining under Model Abliteration](https://mlsanity.com/paper/2510.02768/) (2025)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)

## Abstract (arXiv)

The recent development of Sora leads to a new era in text-to-video (T2V) generation. Along with this comes the rising concern about its security risks. The generated videos may contain illegal or unethical content, and there is a lack of comprehensive quantitative understanding of their safety, posing a challenge to their reliability and practical deployment. Previous evaluations primarily focus on the quality of video generation. While some evaluations of text-to-image models have considered safety, they cover fewer aspects and do not address the unique temporal risk inherent in video generation. To bridge this research gap, we introduce T2VSafetyBench, a new benchmark designed for conducting safety-critical assessments of text-to-video models. We define 12 critical aspects of video generation safety and construct a malicious prompt dataset including real-world prompts, LLM-generated prompts and jailbreak attack-based prompts. Based on our evaluation results, we draw several important findings, including: 1) no single model excels in all aspects, with different models showing various strengths; 2) the correlation between GPT-4 assessments and manual reviews is generally high; 3) there is a trade-off between the usability and safety of text-to-video generative models. This indicates that as the field of video generation rapidly advances, safety risks are set to surge, highlighting the urgency of prioritizing video safety. We hope that T2VSafetyBench can provide insights for better understanding the safety of video generation in the era of generative AI.