mlsanityData to 14 Oct 2026

Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models

arXiv 2510.08625 ↗First posted 8 Oct 2025cs.CVcited by 7

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Despite their impressive generative capabilities, text-to-image diffusion models often memorize and replicate training data, prompting serious concerns over privacy and copyright. Recent work has attributed this memorization to an attraction basin-a region where applying classifier-free guidance (CFG) steers the denoising trajectory toward memorized outputs-and has proposed deferring CFG application until the denoising trajectory escapes this basin. However, such delays often result in non-memorized images that are poorly aligned with the input prompts, highlighting the need to promote earlier escape so that CFG can be applied sooner in the denoising process. In this work, we show that the initial noise sample plays a crucial role in determining when this escape occurs. We empirically observe that different initial samples lead to varying escape times. Building on this insight, we propose two mitigation strategies that adjust the initial noise-either collectively or individually-to find and utilize initial samples that encourage earlier basin escape. These approaches significantly reduce memorization while preserving image-text alignment.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 49 of its 54 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

49 of 54 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Classifier-Free Guidance inside the Attraction Basin May Cause Memorization23 Nov
  2. 2024Exploring Local Memorization in Diffusion Models via Bright Ending Attention29 Oct
  3. 2024Detecting, Explaining, and Mitigating Memorization in Diffusion Models31 Jul
  4. 2024FIND: Fine-tuning Initial Noise Distribution with Policy Optimization for Diffusion Models28 Jul
  5. 2024Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models4 Jun
  6. 2024Towards Memorization-Free Diffusion Models1 Apr
  7. 2024Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention17 Mar
  8. 2023Understanding and Mitigating Copying in Diffusion Models31 May
  9. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidance23 Mar
  10. 2022Classifier-Free Diffusion Guidance26 Jul
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  12. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
5 more
  1. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  3. 2020Denoising Diffusion Implicit Models6 Oct
  4. 2020Denoising Diffusion Probabilistic Models19 Jun
  5. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  8. 2023Extracting Training Data from Diffusion Modelscited by 732
  9. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  11. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 424
  12. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
20 more
  1. 2022A Self-Supervised Descriptor for Image Copy Detectioncited by 176
  2. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  3. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  4. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  5. 2022The Privacy Onion Effect: Memorization is Relativecited by 136
  6. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimizationcited by 123
  7. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  8. 2024D-Flow: Differentiating through Flows for Controlled Generationcited by 122
  9. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 119
  10. 2022Sharpness-Aware Training for Freecited by 95
  11. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  12. 2023Generating images of rare concepts using pre-trained diffusion modelscited by 79
  13. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Modelcited by 63
  14. 2023A Reproducible Extraction of Training Images from Diffusion Modelscited by 61
  15. 2024Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Modelscited by 48
  16. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  17. 2023Norm-guided latent space exploration for text-to-image generationcited by 43
  18. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 28
  19. 2023TIAM -- A Metric for Evaluating Alignment in Text-to-Image Generationcited by 18
  20. 2024Spatial-Aware Latent Initialization for Controllable Image Generationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformerscited by 1
  2. 2026Initialization is Half the Battle: Generating Diverse Images from a Guidance Potential Posteriorcited by 1
  3. 2025CAPTAIN: Semantic Feature Injection for Memorization Mitigation in Text-to-Image Diffusion Modelscited by 0
  4. 2026You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Modelscited by 0
  5. 2026Localizing Memorized Regions in Diffusion Models via Coordinate-Wise Curvature Differencescited by 0
  6. 2026Mitigating Sequential Reappearance in Diffusion Data-Point Unlearningcited by 0
  7. 2026Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Classifier-free Guidance with Adaptive Scaling
  2. 2023Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models
  3. 2025How Diffusion Models Memorize
  4. 2025Navigating with Annealing Guidance Scale in Diffusion Space
  5. 2026Revisiting Classifier-Free Guidance Methods in Latent Diffusion Models
  6. 2025A Minimalist Method for Fine-tuning Text-to-Image Diffusion Models
  7. 2024A Noise is Worth Diffusion Guidance
  8. 2025Noise Projection: Closing the Prompt-Agnostic Gap Behind Text-to-Image Misalignment in Diffusion Models