mlsanityData to 7 Oct 2026

Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models

arXiv 2410.22149 ↗First posted 29 Oct 2024cs.CVcited by 2

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

In this work, we present compelling evidence that controlling model capacity during fine-tuning can effectively mitigate memorization in diffusion models. Specifically, we demonstrate that adopting Parameter-Efficient Fine-Tuning (PEFT) within the pre-train fine-tune paradigm significantly reduces memorization compared to traditional full fine-tuning approaches. Our experiments utilize the MIMIC dataset, which comprises image-text pairs of chest X-rays and their corresponding reports. The results, evaluated through a range of memorization and generation quality metrics, indicate that PEFT not only diminishes memorization but also enhances downstream generation quality. Additionally, PEFT methods can be seamlessly combined with existing memorization mitigation techniques for further improvement. The code for our experiments is available at: https://github.com/Raman1121/Diffusion_Memorization_HPO

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 27 of its 34 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

27 of 34 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Detecting, Explaining, and Mitigating Memorization in Diffusion Models31 Jul
  2. 2023Understanding and Mitigating Copying in Diffusion Models31 May
  3. 2023On the De-duplication of LAION-2B17 Mar
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  5. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  7. 2020Denoising Diffusion Implicit Models6 Oct
  8. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 19

  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2023Extracting Training Data from Diffusion Modelscited by 723
  7. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  8. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  9. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  10. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  11. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  12. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
7 more
  1. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  2. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  3. 2022Three things everyone should know about Vision Transformerscited by 90
  4. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  5. 2020On the limits of cross-domain generalization in automated X-ray predictioncited by 61
  6. 2023Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunitycited by 54
  7. 2023FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysiscited by 23

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Memorization and Regularization in Generative Diffusion Modelscited by 39
  2. 2025The Devil is in the Prompts: De-Identification Traces Enhance Memorization Risks in Synthetic Chest X-Ray Generationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024MemControl: Mitigating Memorization in Diffusion Models via Automated Parameter Selection
  2. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?
  3. 2026On the Memorization of Consistency Distillation for Diffusion Models
  4. 2026Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning
  5. 2025Redistribute Ensemble Training for Mitigating Memorization in Diffusion Models
  6. 2025Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion
  7. 2024Towards Memorization-Free Diffusion Models
  8. 2026Broken Memories: Detecting and Mitigating Memorization in Diffusion Models with Degraded Generations