mlsanityData to 14 Oct 2026

Adynamical systems view of training generativemodels and the memorization phenomenon

arXiv 2605.19483 ↗First posted 19 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Using recent works of one of the authors (VSB) on collapse in generative models and two time scale dynamics in stochastic gradient descent in high dimensions, we give a system theoretic explanation of the memorization phenomenon in generative models. This relies purely on the dynamic aspects of the training phase. Specifically, we use a result of Austin [2016] to motivate a stylized model for the loss function for stochastic gradient descent (SGD) wherein the loss function has a strong dependence on some variables and weak dependence on the rest in a precise sense. This naturally leads to two distinct time scales in the constant step size SGD that is commonly used in machine learning. This fact has been used to explain the double descent phenomenon in SGD in Borkar [2026]. In conjunction with a mathematical model for collapse phenomenon in SGD developed in Borkar [2025a], we analyze the constant step size SGD using the recent results of Azizian et al. [2024] in order to explain the phenomenon of memorization wherein a generative model that is concurrently being tuned yields the same or similar outputs for significant stretches of time. This gives a novel perspective on the aforementioned phenomena reported in machine learning literature and their interrelationships, using a dynamical systems viewpoint.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 48 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

21 of 48 references.

References · 21

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  2. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  3. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  4. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  5. 2023Self-Consuming Generative Models Go MADcited by 250
  6. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  7. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
  8. 2024Detecting, Explaining, and Mitigating Memorization in Diffusion Modelscited by 95
  9. 2024Model Collapse Demystified: The Case of Regressioncited by 82
  10. 2024Towards Memorization-Free Diffusion Modelscited by 50
  11. 2025Memorization and Regularization in Generative Diffusion Modelscited by 40
  12. 2024Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Modelscited by 37
9 more
  1. 2025Memorization to Generalization: Emergence of Diffusion Models from Associative Memorycited by 36
  2. 2024Rate of Model Collapse in Recursive Trainingcited by 25
  3. 2024Heat Death of Generative Models in Closed-Loop Learningcited by 22
  4. 2025On the Edge of Memorization in Diffusion Modelscited by 22
  5. 2024What is the long-run distribution of stochastic gradient descent? A large deviations analysiscited by 17
  6. 2025Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorizationcited by 17
  7. 2025Provable Separations between Memorization and Generalization in Diffusion Modelscited by 14
  8. 2025A dynamic view of some anomalous phenomena in SGDcited by 2
  9. 2025A theoretical basis for model collapse in recursive trainingcited by 1

Built on by

Not cited yet.