mlsanityData to 7 Oct 2026

SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset

arXiv 2406.14477 ↗First posted 20 Jun 2024cs.CV cs.AI cs.DBcited by 34

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

To mitigate the risk of harmful outputs from large vision models (LVMs), we introduce the SafeSora dataset to promote research on aligning text-to-video generation with human values. This dataset encompasses human preferences in text-to-video generation tasks along two primary dimensions: helpfulness and harmlessness. To capture in-depth human preferences and facilitate structured reasoning by crowdworkers, we subdivide helpfulness into 4 sub-dimensions and harmlessness into 12 sub-categories, serving as the basis for pilot annotations. The SafeSora dataset includes 14,711 unique prompts, 57,333 unique videos generated by 4 distinct LVMs, and 51,691 pairs of preference annotations labeled by humans. We further demonstrate the utility of the SafeSora dataset through several applications, including training the text-video moderation model and aligning LVMs with human preference by fine-tuning a prompt augmentation module or the diffusion model. These applications highlight its potential as the foundation for text-to-video alignment research, such as human preference modeling and the development and validation of alignment algorithms.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 95 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

41 of 95 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
29 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  3. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  6. 2023ModelScope Text-to-Video Technical Reportcited by 651
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  8. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  9. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  10. 2016Tutorial on Variational Autoencoderscited by 404
  11. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  12. 2016Generating Videos with Scene Dynamicscited by 376
  13. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  14. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  15. 2021Unsolved Problems in ML Safetycited by 291
  16. 2023Model evaluation for extreme riskscited by 167
  17. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  18. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 128
  19. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  20. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  21. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  22. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  23. 2024Vlogger: Make Your Dream A Vlogcited by 70
  24. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  25. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videoscited by 51
  26. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  27. 2024A Survey of Imitation Learning Methods, Environments and Metricscited by 16
  28. 2023Bitstream-Corrupted Video Recovery: A Novel Benchmark Dataset and Methodcited by 14
  29. 2021VidHarm: A Clip Based Dataset for Harmful Content Detectioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 13.

34 papers.

Sort:

By area:

  1. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  2. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  3. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  4. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 25
  5. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generationcited by 21
  6. 2025T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Modelcited by 16
  7. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 14
  8. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  9. 2025T2VUnlearning: A Concept Erasing Method for Text-to-Video Diffusion Modelscited by 12
  10. 2025VideoEraser: Concept Erasure in Text-to-Video Diffusion Modelscited by 12
  11. 2025Video Unlearning via Low-Rank Refusal Vectorcited by 11
  12. 2025Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Modelscited by 10
22 more
  1. 2025GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoningcited by 7
  2. 2026Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarkscited by 6
  3. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  4. 2025PurifyGen: A Risk-Discrimination and Semantic-Purification Model for Safe Text-to-Image Generationcited by 4
  5. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 3
  6. 2025CGCE: Classifier-Guided Concept Erasure in Generative Modelscited by 3
  7. 2025Now You See It, Now You Don't - Instant Concept Erasure for Safe Text-to-Image and Video Generationcited by 3
  8. 2026SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generationcited by 3
  9. 2026Activation Steering of Video Generation Models via Reduced-Order Linear Optimal Controlcited by 2
  10. 2025AVFakeBench: A Comprehensive Audio-Video Forgery Detection Benchmark for AV-LMMscited by 1
  11. 2025On Monotonicity in AI Alignmentcited by 0
  12. 2025HKGAI-V1: Towards Regional Sovereign Large Language Model for Hong Kongcited by 0
  13. 2026DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generationcited by 0
  14. 2026Prompt-to-Gesture: Measuring the Capabilities of Image-to-Video Deictic Gesture Generationcited by 0
  15. 2026PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulationcited by 0
  16. 2026Pulling The REINS: Training-Free Safety Alignment of Video Diffusion Models via Representation Steeringcited by 0
  17. 2026Uni-AdaVD: Universal Concept Erasure for Visual Generation via Orthogonal Value Decompositioncited by 0
  18. 2026Multi2AV-Safety: Benchmarking Safety in Multimodal-to-Audio-Video Generationcited by 0
  19. 2026Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderationcited by 0
  20. 2026AV-SafetyBench: A Safety Benchmark for Text-to-Audio-Video Generationcited by 0
  21. 2026CleanVideo: Adaptive Concept Erasure for Text-to-Video Diffusion Modelscited by 0
  22. 2026TempQ-Jail: Query-Constrained Candidate Ranking for Text-to-Video Jailbreak Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generation
  2. 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment
  3. 2025Personalized Safety Alignment for Text-to-Image Diffusion Models
  4. 2025SafeVid: Toward Safety Aligned Video Large Multimodal Models
  5. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
  6. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models
  7. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
  8. 2025HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignment