# The Diffusion Encoder

arXiv: https://arxiv.org/abs/2605.13399 · HTML: https://mlsanity.com/paper/2605.13399/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 13 May 2026.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control](https://mlsanity.com/paper/2409.08861/) (2024)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Interacting particle solutions of Fokker-Planck equations through gradient-log-density estimation](https://mlsanity.com/paper/2006.00702/) (2020)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)
- [DIME:Diffusion-Based Maximum Entropy Reinforcement Learning](https://mlsanity.com/paper/2502.02316/) (2025)
- [Information-Theoretic Diffusion](https://mlsanity.com/paper/2302.03792/) (2023)
- [Stochastic Optimal Control Matching](https://mlsanity.com/paper/2312.02027/) (2023)
- [Interpretable Diffusion via Information Decomposition](https://mlsanity.com/paper/2310.07972/) (2023)
- [Q-learning with Adjoint Matching](https://mlsanity.com/paper/2601.14234/) (2026)
- [MINDE: Mutual Information Neural Diffusion Estimation](https://mlsanity.com/paper/2310.09031/) (2023)
- [Neural Entropy](https://mlsanity.com/paper/2409.03817/) (2024)
- [On the Separability of Information in Diffusion Models](https://mlsanity.com/paper/2509.23937/) (2025)

## Abstract (arXiv)

We construct a new kind of encoder, leveraging the expressive power of diffusion models. In a traditional variational autoencoder, the encoder and decoder jointly negotiate a latent representation of the input. This is made possible by the reparameterization trick, which simplifies training at the cost of restricting the encoder to a simple family of distributions. Replacing this encoder with a diffusion model requires rethinking how the decoder pressure can be transmitted back to the encoder, given that they tend to update their internal estimates of the latent in opposing directions. We solve this problem with an alternating training scheme, inspired by the expectation-maximization algorithm. Our method enables more reliable synchronization between encoder and decoder, while preserving the simple and efficient training objective of standard diffusion models.