# Diffusion As Self-Distillation: End-to-End Latent Diffusion In One Model

arXiv: https://arxiv.org/abs/2511.14716 · HTML: https://mlsanity.com/paper/2511.14716/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 8 of its 50 references that are in the map.
- First posted on arXiv 18 Nov 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 8 of 8)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Diffusion Transformers with Representation Autoencoders](https://mlsanity.com/paper/2510.11690/) (2025)
- [MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer](https://mlsanity.com/paper/2303.14389/) (2023)
- [Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization](https://mlsanity.com/paper/2503.11056/) (2025)
- [Latent Denoising Makes Good Tokenizers](https://mlsanity.com/paper/2507.15856/) (2025)

## Built on by (most cited first, 3 of 3)

- [CrossFlow: One-Step Generation Across Latent and Pixel Spaces](https://mlsanity.com/paper/2606.19970/) (2026)
- [SPROUT: A Scalable Diffusion Foundation Model for Multi-Crop Plant Phenotyping](https://mlsanity.com/paper/2603.27519/) (2026)
- [LDM-is-AE: Latent Diffusion Model is an Auto-Encoder for End-to-End Image Generation](https://mlsanity.com/paper/2609.37080/) (2026)

## Abstract (arXiv)

Standard Latent Diffusion Models rely on a complex, three-part architecture consisting of a separate encoder, decoder, and diffusion network, which are trained in multiple stages. This modular design is computationally inefficient, leads to suboptimal performance, and prevents the unification of diffusion with the single-network architectures common in vision foundation models. Our goal is to unify these three components into a single, end-to-end trainable network. We first demonstrate that a naive joint training approach fails catastrophically due to ``latent collapse'', where the diffusion training objective interferes with the network's ability to learn a good latent representation. We identify the root causes of this instability by drawing a novel analogy between diffusion and self-distillation based unsupervised learning method. Based on this insight, we propose Diffusion as Self-Distillation (DSD), a new framework with key modifications to the training objective that stabilize the latent space. This approach enables, for the first time, the stable end-to-end training of a single network that simultaneously learns to encode, decode, and perform diffusion. DSD achieves outstanding performance on the ImageNet 256× 256 conditional generation task: FID=13.44/6.38/4.25 with only 42M/118M/205M parameters and 50 training epochs on ImageNet, without using classifier-free-guidance.