# Self-Improving Diffusion Models with Synthetic Data

arXiv: https://arxiv.org/abs/2408.16333 · HTML: https://mlsanity.com/paper/2408.16333/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [Guiding a Diffusion Model with a Bad Version of Itself](https://mlsanity.com/paper/2406.02507/) (2024)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [Scalable Adaptive Computation for Iterative Generation](https://mlsanity.com/paper/2212.11972/) (2022)
- [Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness](https://mlsanity.com/paper/2302.10893/) (2023)
- [Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data](https://mlsanity.com/paper/2404.01413/) (2024)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance](https://mlsanity.com/paper/2403.17377/) (2024)
- [End-to-End Diffusion Latent Optimization Improves Classifier Guidance](https://mlsanity.com/paper/2303.13703/) (2023)
- [Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models](https://mlsanity.com/paper/2211.17091/) (2022)
- [On the Stability of Iterative Retraining of Generative Models on their own Data](https://mlsanity.com/paper/2310.00429/) (2023)
- [RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold](https://mlsanity.com/paper/2406.14532/) (2024)
- [Model Collapse Demystified: The Case of Regression](https://mlsanity.com/paper/2402.07712/) (2024)
- [RoSteALS: Robust Steganography using Autoencoder Latent Space](https://mlsanity.com/paper/2304.03400/) (2023)
- [Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias](https://mlsanity.com/paper/2403.07857/) (2024)
- [TrustMark: Universal Watermarking for Arbitrary Resolution Images](https://mlsanity.com/paper/2311.18297/) (2023)
- [Self-Correcting Self-Consuming Loops for Generative Model Training](https://mlsanity.com/paper/2402.07087/) (2024)
- [Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values](https://mlsanity.com/paper/2203.01993/) (2022)

## Built on by (most cited first, 30 of 35)

- [Strong Model Collapse](https://mlsanity.com/paper/2410.04840/) (2024)
- [NoHumansRequired: Autonomous High-Quality Image Editing Triplet Mining](https://mlsanity.com/paper/2507.14119/) (2025)
- [Universality of the π2/6 Pathway in Avoiding Model Collapse](https://mlsanity.com/paper/2410.22812/) (2024)
- [Self-Guidance: Boosting Flow and Diffusion Generation on Their Own](https://mlsanity.com/paper/2412.05827/) (2024)
- [Ambient Diffusion Omni: Training Good Models with Bad Data](https://mlsanity.com/paper/2506.10038/) (2025)
- [A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective](https://mlsanity.com/paper/2509.16499/) (2025)
- [A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops](https://mlsanity.com/paper/2502.18865/) (2025)
- [Machine-generated text detection prevents language model collapse](https://mlsanity.com/paper/2502.15654/) (2025)
- [EarthSynth: Generating Informative Earth Observation with Diffusion Models](https://mlsanity.com/paper/2505.12108/) (2025)
- [Delving into: the quantification of Ai-generated content on the internet (synthetic data)](https://mlsanity.com/paper/2504.08755/) (2025)
- [Self-Consuming Generative Models with Adversarially Curated Data](https://mlsanity.com/paper/2505.09768/) (2025)
- [PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Models](https://mlsanity.com/paper/2503.10529/) (2025)
- [Why Less is More (Sometimes): A Theory of Data Curation](https://mlsanity.com/paper/2511.03492/) (2025)
- [Self-Improving VLM Judges Without Human Annotations](https://mlsanity.com/paper/2512.05145/) (2025)
- [Increasing the Utility of Synthetic Images through Chamfer Guidance](https://mlsanity.com/paper/2508.10631/) (2025)
- [Neon: Negative Extrapolation From Self-Training Improves Image Generation](https://mlsanity.com/paper/2510.03597/) (2025)
- [Stabilizing Self-Consuming Diffusion Models with Latent Space Filtering](https://mlsanity.com/paper/2511.12742/) (2025)
- [Quantifying Error Propagation and Model Collapse in Diffusion Models](https://mlsanity.com/paper/2602.16601/) (2026)
- [Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling](https://mlsanity.com/paper/2501.04666/) (2025)
- [Active Flow Expansion for Out-of-Distribution Discovery: from Theory to Molecules](https://mlsanity.com/paper/2606.08802/) (2026)
- [Guiding a diffusion model using sliding windows](https://mlsanity.com/paper/2411.10257/) (2024)
- [Noise-Level Diffusion Guidance: Well Begun is Half Done](https://mlsanity.com/paper/2509.13936/) (2025)
- [Beyond Real Data: Synthetic Data through the Lens of Regularization](https://mlsanity.com/paper/2510.08095/) (2025)
- [Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loop](https://mlsanity.com/paper/2601.05184/) (2026)
- [Ambient Dataloops: Generative Models for Dataset Refinement](https://mlsanity.com/paper/2601.15417/) (2026)
- [ForTIFAI: Fending Off Recursive Training Induced Failure for AI Model Collapse](https://mlsanity.com/paper/2509.08972/) (2025)
- [Overcoming the Curvature Bottleneck in MeanFlow](https://mlsanity.com/paper/2511.23342/) (2025)
- [Not All Synthetic Data Is Yours to Learn From](https://mlsanity.com/paper/2605.31126/) (2026)
- [ReSAGE-PAR: Representational Similarity Assessment for Generative Expansion in Pedestrian Attribute Recognition](https://mlsanity.com/paper/2606.06020/) (2026)
- [Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterization](https://mlsanity.com/paper/2606.13796/) (2026)

## Abstract (arXiv)

The artificial intelligence (AI) world is running out of real data for training increasingly large generative models, resulting in accelerating pressure to train on synthetic data. Unfortunately, training new generative models with synthetic data from current or past generation models creates an autophagous (self-consuming) loop that degrades the quality and/or diversity of the synthetic data in what has been termed model autophagy disorder (MAD) and model collapse. Current thinking around model autophagy recommends that synthetic data is to be avoided for model training lest the system deteriorate into MADness. In this paper, we take a different tack that treats synthetic data differently from real data. Self-IMproving diffusion models with Synthetic data (SIMS) is a new training concept for diffusion models that uses self-synthesized data to provide negative guidance during the generation process to steer a model's generative process away from the non-ideal synthetic data manifold and towards the real data distribution. We demonstrate that SIMS is capable of self-improvement; it establishes new records based on the Fr'echet inception distance (FID) metric for CIFAR-10 and ImageNet-64 generation and achieves competitive results on FFHQ-64 and ImageNet-512. Moreover, SIMS is, to the best of our knowledge, the first prophylactic generative AI algorithm that can be iteratively trained on self-generated synthetic data without going MAD. As a bonus, SIMS can adjust a diffusion model's synthetic data distribution to match any desired in-domain target distribution to help mitigate biases and ensure fairness.