mlsanityData to 14 Oct 2026

Elucidating Representation Degradation Problem in Diffusion Model Training

arXiv 2605.10790 ↗First posted 11 May 2026cs.LGcited by 0

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models have achieved remarkable success, yet their training remains inefficient due to a severe optimization bottleneck, which we term Representation Degradation. As noise levels increase, the outputs of the trained model exhibit progressive structural distortion, which can destabilize training and impair generation quality. Our analysis suggests that this instability is driven by mismatched target recoverability, which is associated with Neural Tangent Kernel (NTK) spectral weakening and effective low-rank behavior. To address this, we propose Elucidated Representation Diffusion (ERD), a plug-and-play framework that dynamically reallocates optimization effort according to effective recoverability. By stabilizing representation learning without external supervision, ERD accelerates convergence and achieves strong empirical performance across diffusion backbones.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 59 of its 65 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

59 of 65 references.

Closest ancestors · 39

Same direction, newest first

  1. 2025Latent Diffusion Model without Variational Autoencoder17 Oct
  2. 2025Diffusion Transformers with Representation Autoencoders13 Oct
  3. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think2 Jul
  4. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves5 May
  5. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers14 Apr
  6. 2025USP: Unified Self-Supervised Pretraining for Image Generation and Understanding8 Mar
  7. 2025Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling9 Feb
  8. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models2 Jan
  9. 2024FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification14 Oct
  10. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think9 Oct
  11. 2024Improved Noise Schedule for Diffusion Training3 Jul
  12. 2024A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training27 May
27 more
  1. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models11 Apr
  2. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer25 Mar
  3. 2024Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon14 Mar
  4. 2024Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization28 Jan
  5. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  6. 2023Analyzing and Improving the Training Dynamics of Diffusion Models5 Dec
  7. 2023DiffiT: Diffusion Vision Transformers for Image Generation4 Dec
  8. 2023Unmasking Bias in Diffusion Model Training12 Oct
  9. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  10. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  11. 2023Efficient Diffusion Training via Min-SNR Weighting Strategy16 Mar
  12. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation1 Mar
  13. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  14. 2022Scalable Diffusion Models with Transformers19 Dec
  15. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  16. 2022Understanding Diffusion Models: A Unified Perspective25 Aug
  17. 2022Classifier-Free Diffusion Guidance26 Jul
  18. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  19. 2022Perception Prioritized Training of Diffusion Models1 Apr
  20. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  21. 2021Variational Diffusion Models1 Jul
  22. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  23. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  24. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  25. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  26. 2020Denoising Diffusion Implicit Models6 Oct
  27. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  8. 2016Improved Techniques for Training GANscited by 2,970
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2022Video Diffusion Modelscited by 2,305
  11. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  12. 2018On the Spectral Bias of Neural Networkscited by 1,341
8 more
  1. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  2. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  4. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  5. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  6. 2021Generating Images with Sparse Representationscited by 266
  7. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  8. 2023Intriguing Properties of Quantization at Scalecited by 49

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterization
  2. 2026Residual-Stream Burden Shapes Representation Learning in Diffusion Transformers
  3. 2025Learning Diffusion Models with Flexible Representation Guidance
  4. 2025Diagnosing and Improving Diffusion Models by Estimating the Optimal Loss Value
  5. 2025Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deployment
  6. 2026Revisiting Spectral Representations in Generative Diffusion Models
  7. 2025Diffusion Models: A Mathematical Introduction
  8. 2025Generalization of Diffusion Models Arises with a Balanced Representation Space