mlsanityData to 14 Oct 2026

Memorization In Stable Diffusion Is Unexpectedly Driven by CLIP Embeddings

arXiv 2605.02908 ↗First posted 6 Apr 2026cs.CV cs.AI cs.LGcited by 0

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Understanding how textual embeddings contribute to memorization in text-to-image diffusion models is crucial for both interpretability and safety. This paper investigates an unexpected behavior of CLIP embeddings in Stable Diffusion, revealing that the model disproportionately relies on specific embeddings. We categorize input tokens as <startoftext>, <prompt>, <endoftext> and <pad> with corresponding embeddings vsot, vpr, veot, vpad. We discover that vpr contribute minimally to generation in memorized cases. In contrast, vpad strongly affect memorization due to their structural duplication of veot, the only embedding explicitly optimized during CLIP training. This duplication unintentionally amplifies the influence of veot, causing the model to over-rely on it, thereby driving memorization. Based on these observations, we propose two simple yet effective inference-time mitigation strategies: (1) Replacing the tokenizer's default <pad> from <eot> to the ! token before embedding, and masking the veot; (2) Partial masking of vpad. Both suppress memorization without degrading quality, and are readily deployable without prior detection.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 32 of its 37 references that are in the map. First posted on arXiv 6 Apr 2026.

Builds on

32 of 37 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025How Diffusion Models Memorize30 Sep
  2. 2024Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapes5 Dec
  3. 2024Exploring Local Memorization in Diffusion Models via Bright Ending Attention29 Oct
  4. 2024Detecting, Explaining, and Mitigating Memorization in Diffusion Models31 Jul
  5. 2024MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models24 Jul
  6. 2024Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models4 Jun
  7. 2024Towards Memorization-Free Diffusion Models1 Apr
  8. 2024Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data20 Mar
  9. 2024Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention17 Mar
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  11. 2023On Memorization in Diffusion Models4 Oct
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
5 more
  1. 2023Understanding and Mitigating Copying in Diffusion Models31 May
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  3. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  4. 2020Denoising Diffusion Implicit Models6 Oct
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  8. 2023Extracting Training Data from Diffusion Modelscited by 732
  9. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 424
  10. 2022A Self-Supervised Descriptor for Image Copy Detectioncited by 176
  11. 2023Ambient Diffusion: Learning Clean Distributions from Corrupted Datacited by 111
  12. 2023A Reproducible Extraction of Training Images from Diffusion Modelscited by 61
3 more
  1. 2024Towards Understanding the Working Mechanism of Text-to-Image Diffusion Modelcited by 48
  2. 2024A Geometric Framework for Understanding Memorization in Generative Modelscited by 40
  3. 2025Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Modelscited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Memorized Images in Diffusion Models share a Subspace that can be Located and Deleted
  2. 2025Finding DoRI: Discovery of Retained Images in Diffusion Models
  3. 2026Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning
  4. 2023Understanding (Un)Intended Memorization in Text-to-Image Generative Models
  5. 2025CLIP is All You Need for Human-like Semantic Representations in Stable Diffusion
  6. 2023Diversity and Diffusion: Observations on Synthetic Image Distributions with Stable Diffusion
  7. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attention
  8. 2025Generalization of Diffusion Models Arises with a Balanced Representation Space