mlsanityData to 14 Oct 2026

AlignGuard: Scalable Safety Alignment for Text-to-Image Generation

arXiv 2412.10493 ↗First posted 13 Dec 2024cs.CV cs.AI cs.LGcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

also in 1 more
Abstract (arXiv)

Text-to-image (T2I) models are widespread, but their limited safety guardrails expose end users to harmful content and potentially allow for model misuse. Current safety measures are typically limited to text-based filtering or concept removal strategies, able to remove just a few concepts from the model's generative capabilities. In this work, we introduce AlignGuard, a method for safety alignment of T2I models. We enable the application of Direct Preference Optimization (DPO) for safety purposes in T2I models by synthetically generating a dataset of harmful and safe image-text pairs, which we call CoProV2. Using a custom DPO strategy and this dataset, we train safety experts, in the form of low-rank adaptation (LoRA) matrices, able to guide the generation process away from specific safety-related concepts. Then, we merge the experts into a single LoRA using a novel merging strategy for optimal scaling performance. This expert-based approach enables scalability, allowing us to remove 7x more harmful concepts from T2I models compared to baselines. AlignGuard consistently outperforms the state-of-the-art on many benchmarks and establishes new practices for safety alignment in T2I networks. Code and data will be shared at https://safetydpo.github.io/.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 62 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

53 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization13 Mar
  2. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov

Other references · 51

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
38 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  2. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  3. 2022Editing Models with Task Arithmeticcited by 1,120
  4. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  5. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  6. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  7. 2023Erasing Concepts from Diffusion Modelscited by 539
  8. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  9. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  10. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  11. 2023Unified Concept Editing in Diffusion Modelscited by 388
  12. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  13. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  14. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  15. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 317
  16. 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 307
  17. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 301
  18. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 264
  19. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 250
  20. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  21. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  22. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  23. 2023MMA-Diffusion: MultiModal Attack on Diffusion Modelscited by 194
  24. 2023SneakyPrompt: Jailbreaking Text-to-image Generative Modelscited by 189
  25. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 175
  26. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Promptscited by 171
  27. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 155
  28. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 130
  29. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  30. 2022Fusing finetuned models for better pretrainingcited by 110
  31. 2024Multi-LoRA Composition for Image Generationcited by 80
  32. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 71
  33. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
  34. 2020Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Wellcited by 57
  35. 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 55
  36. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 43
  37. 2024Near to Mid-term Risks and Opportunities of Open-Source Generative AIcited by 13
  38. 2024Risks and Opportunities of Open-Source Generative AIcited by 13

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 11.

20 papers.

Sort:

By area:

  1. 2025PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Modelscited by 17
  2. 2025Erasing Concepts, Steering Generations: A Comprehensive Survey of Concept Suppressioncited by 10
  3. 2025Erased but Not Forgotten: How Backdoors Compromise Concept Erasurecited by 9
  4. 2025Red-Teaming Text-to-Image Systems by Rule-based Preference Modelingcited by 8
  5. 2025DeContext as Defense: Safe Image Editing in Diffusion Transformerscited by 7
  6. 2026GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Promptcited by 7
  7. 2025OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Modelscited by 5
  8. 2025DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modelingcited by 5
  9. 2025Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Surveycited by 3
  10. 2025SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generationcited by 2
  11. 2026Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Modelscited by 1
  12. 2026Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairscited by 1
8 more
  1. 2026The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Modelscited by 1
  2. 2026SPOT: Selective Prompt Projection via Total Variation for Inference-Only Safe Text-to-Image Generationcited by 0
  3. 2026SafeCtrl: Region-Aware Safety Control for Text-to-Image Diffusion via Detect-Then-Suppresscited by 0
  4. 2026SafeDiffusion-R1: Online Reward Steering for Safe Diffusion Post-Trainingcited by 0
  5. 2026Automatic Red Teaming for Implicit Vulnerabilities of Text-to-Image Modelscited by 0
  6. 2026FestDPO: Few-step Generator Alignment with Direct Preference Optimizationcited by 0
  7. 2026Keep It CALM: Analyzing the Limits of Global Unsafety in Text-to-Image Generationcited by 0
  8. 2026SaTPO: Text-to-Image Safety Alignment via Transition Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Path to Reconciling Quality and Safety in Text-to-Image Generation: Dataset, Method, and Evaluation
  2. 2025SPQR: A Multi-Dimensional Benchmark for Safety Alignment under Benign Model Adaptation
  3. 2024Universal Prompt Optimizer for Safe Text-to-Image Generation
  4. 2025Personalized Safety Alignment for Text-to-Image Diffusion Models
  5. 2025SafeText: Safe Text-to-image Models via Aligning the Text Encoder
  6. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation
  7. 2025SafeEditor: Unified MLLM for Efficient Post-hoc T2I Safety Editing
  8. 2025YINYANG-ALIGN: Benchmarking Contradictory Objectives and Proposing Multi-Objective Optimization based DPO for Text-to-Image Alignment