mlsanityData to 13 Oct 2026

Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation

arXiv 2311.17216 ↗First posted 28 Nov 2023cs.CVcited by 69

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion-based models have gained significant popularity for text-to-image generation due to their exceptional image-generation capabilities. A risk with these models is the potential generation of inappropriate content, such as biased or harmful images. However, the underlying reasons for generating such undesired content from the perspective of the diffusion model's internal representation remain unclear. Previous work interprets vectors in an interpretable latent space of diffusion models as semantic concepts. However, existing approaches cannot discover directions for arbitrary concepts, such as those related to inappropriate concepts. In this work, we propose a novel self-supervised approach to find interpretable latent directions for a given concept. With the discovered vectors, we further propose a simple approach to mitigate inappropriate generation. Extensive experiments have been conducted to verify the effectiveness of our mitigation approach, namely, for fair generation, safe generation, and responsible text-enhancing generation. Project page: urlhttps://interpretdiffusion.github.io.

Key facts. Cited by 69 papers in the map (data through 13 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 48 of its 55 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

48 of 55 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Models14 Mar
  2. 2023Cones: Concept Neurons in Diffusion Models for Customized Generation9 Mar
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  4. 2022Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models16 Dec
  5. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  7. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,656
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,028
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,023
  8. 2022Scalable Diffusion Models with Transformerscited by 5,973
  9. 2022Classifier-Free Diffusion Guidancecited by 5,785
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,023
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,421
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,921
29 more
  1. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 903
  2. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 658
  3. 2023Erasing Concepts from Diffusion Modelscited by 535
  4. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 509
  5. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 448
  6. 2023Unified Concept Editing in Diffusion Modelscited by 384
  7. 2022Diffusion Models already have a Semantic Latent Spacecited by 325
  8. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 316
  9. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 301
  10. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 263
  11. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  12. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 210
  13. 2023FreeU: Free Lunch in Diffusion U-Netcited by 196
  14. 2023Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Modelscited by 177
  15. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 154
  16. 2023Fair Diffusion: Instructing Text-to-Image Generation Models on Fairnesscited by 150
  17. 2023Debiasing Vision-Language Models via Biased Promptscited by 148
  18. 2023Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometrycited by 122
  19. 2023ITI-GEN: Inclusive Text-to-Image Generationcited by 64
  20. 2023Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Modelscited by 60
  21. 2022Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Modelscited by 54
  22. 2023InfoDiffusion: Representation Learning Using Information Maximizing Diffusion Modelscited by 49
  23. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  24. 2023Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusioncited by 27
  25. 2023Mitigating Inappropriateness in Image Generation: Can there be Value in Reflecting the World's Ugliness?cited by 16
  26. 2023ORES: Open-vocabulary Responsible Visual Synthesiscited by 14
  27. 2022Do DALL-E and Flamingo Understand Each Other?cited by 7
  28. 2023Towards Robust Prompts on Vision-Language Modelscited by 7
  29. 2023DiffuseGAE: Controllable and High-fidelity Image Manipulation from Disentangled Representationcited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 21; 2025: 29; 2026: 19.

69 papers.

Sort:

By area:

  1. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 130
  2. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 70
  3. 2024UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Imagescited by 53
  4. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 40
  5. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 32
  6. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  7. 2025ShieldGemma 2: Robust and Tractable Image Content Moderationcited by 28
  8. 2024Separable Multi-Concept Erasure from Diffusion Modelscited by 27
  9. 2025CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Modelscited by 27
  10. 2024How to Trace Latent Generative Model Generated Images without Artificial Watermark?cited by 26
  11. 2024TraSCE: Trajectory Steering for Concept Erasurecited by 26
  12. 2024Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusioncited by 22
38 more
  1. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 19
  2. 2024Which Model Generated This Image? A Model-Agnostic Approach for Origin Attributioncited by 16
  3. 2025Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimizationcited by 13
  4. 2024Don't Let Your Robot be Harmful: Responsible Robotic Manipulation via Safety-as-Policycited by 9
  5. 2025Hessian Geometry of Latent Space in Generative Modelscited by 9
  6. 2025Fair Generation without Unfair Distortions: Debiasing Text-to-Image Generation with Entanglement-Free Attentioncited by 9
  7. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  8. 2024FairGen: Enhancing Fairness in Text-to-Image Diffusion Models via Self-Discovering Latent Directionscited by 8
  9. 2025CE-SDWV: Effective and Efficient Concept Erasure for Text-to-Image Diffusion Models via a Semantic-Driven Word Vocabularycited by 8
  10. 2024Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Modelscited by 7
  11. 2024AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Modelscited by 7
  12. 2025TarPro: Targeted Protection against Malicious Image Editingcited by 7
  13. 2025FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Modelscited by 7
  14. 2025AutoMV: An Automatic Multi-Agent System for Music Video Generationcited by 7
  15. 2024Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Surveycited by 6
  16. 2025Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Controlcited by 5
  17. 2025Concept Lancet: Image Editing with Compositional Representation Transplantcited by 4
  18. 2025Now You See It, Now You Don't - Instant Concept Erasure for Safe Text-to-Image and Video Generationcited by 4
  19. 2026Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Modelscited by 4
  20. 2024Exploring the Potentials and Challenges of Deep Generative Models in Product Design Conceptioncited by 3
  21. 2025Exploring the latent space of diffusion models directly through singular value decompositioncited by 3
  22. 2025Superscopes: Amplifying Internal Feature Representations for Language Model Interpretationcited by 3
  23. 2025RespoDiff: Dual-Module Bottleneck Transformation for Responsible & Faithful T2I Generationcited by 3
  24. 2025Fully Unsupervised Self-debiasing of Text-to-Image Diffusion Modelscited by 3
  25. 2025UniLight: A Unified Representation for Lightingcited by 3
  26. 2025Rethinking the Vulnerability of Concept Erasure and a New Methodcited by 2
  27. 2025Mind the Trojan Horse: Image Prompt Adapter Enabling Scalable and Deceptive Jailbreakingcited by 2
  28. 2025An h-space Based Adversarial Attack for Protection Against Few-shot Personalizationcited by 2
  29. 2025Beyond the Safety Tax: Mitigating Unsafe Text-to-Image Generation via External Safety Rectificationcited by 2
  30. 2025SCALEX: Scalable Concept and Latent Exploration for Diffusion Modelscited by 2
  31. 2026Selective Fine-Tuning for Targeted and Robust Concept Unlearningcited by 2
  32. 2026Locating Demographic Bias at the Attention-Head Level in CLIP's Vision Encodercited by 2
  33. 2026Attention, May I Have Your Decision? Localizing Generative Choices in Diffusion Modelscited by 2
  34. 2026Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generationcited by 2
  35. 2024Safety Without Semantic Disruptions: Editing-free Safe Image Generation via Context-preserving Dual Latent Reconstructioncited by 1
  36. 2026FAIRT2V: Training-Free Debiasing for Text-to-Video Diffusion Modelscited by 1
  37. 2026Encoding Predictability and Legibility for Style-Conditioned Diffusion Policycited by 1
  38. 2026EGLOCE: Training-Free Energy-Guided Latent Optimization for Concept Erasurecited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Emergence and Evolution of Interpretable Concepts in Diffusion Models
  2. 2023The Hidden Language of Diffusion Models
  3. 2024Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Prompts
  4. 2025Responsible Diffusion Models via Constraining Text Embeddings within Safe Regions
  5. 2023NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models
  6. 2024Unveiling Concept Attribution in Diffusion Models
  7. 2023Trade-offs in Fine-tuned Diffusion Models Between Accuracy and Interpretability
  8. 2026Localized Concept Erasure in Text-to-Image Diffusion Models via High-Level Representation Misdirection