mlsanityData to 14 Oct 2026

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

arXiv 2111.15640 ↗First posted 30 Nov 2021cs.CV cs.LGcited by 449

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion probabilistic models (DPMs) have achieved remarkable quality in image generation that rivals GANs'. But unlike GANs, DPMs use a set of latent variables that lack semantic meaning and cannot serve as a useful representation for other tasks. This paper explores the possibility of using DPMs for representation learning and seeks to extract a meaningful and decodable representation of an input image via autoencoding. Our key idea is to use a learnable encoder for discovering the high-level semantics, and a DPM as the decoder for modeling the remaining stochastic variations. Our method can encode any image into a two-part latent code, where the first part is semantically meaningful and linear, and the second part captures stochastic details, allowing near-exact reconstruction. This capability enables challenging applications that currently foil GAN-based methods, such as attribute manipulation on real images. We also show that this two-level encoding improves denoising efficiency and naturally facilitates various downstream tasks including few-shot conditional sampling. Please visit our project page: https://Diff-AE.github.io/

Key facts. Cited by 449 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 56 of its 66 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

56 of 66 references.

Closest ancestors · 12

Same direction, newest first

  1. 2021Bilateral Denoising Diffusion Models26 Aug
  2. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models6 Aug
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  4. 2021Variational Diffusion Models1 Jul
  5. 2021Diffusion Priors In Variational Autoencoders29 Jun
  6. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  7. 2021Gotta Go Fast When Generating Data with Score-Based Models28 May
  8. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  9. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  11. 2020Denoising Diffusion Implicit Models6 Oct
  12. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  11. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
32 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  2. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  3. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  4. 2018Group Normalizationcited by 1,524
  5. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  6. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  7. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  8. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  10. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  11. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  12. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  13. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 735
  14. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  15. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 695
  16. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  17. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  18. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  19. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  20. 2020GANSpace: Discovering Interpretable GAN Controlscited by 614
  21. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  22. 2021GAN Inversion: A Surveycited by 367
  23. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  24. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 349
  25. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
  26. 2017AttGAN: Facial Attribute Editing by Only Changing What You Wantcited by 242
  27. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  28. 2021D2C: Diffusion-Denoising Models for Few-shot Conditional Generationcited by 96
  29. 2020Simple and Effective VAE Training with Calibrated Decoderscited by 57
  30. 2019RelGAN: Multi-Domain Image-to-Image Translation via Relative Attributescited by 56
  31. 2018Distribution Matching in Variational Inferencecited by 35
  32. 2019Learning to Predict Without Looking Ahead: World Models Without Forward Predictioncited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 37; 2023: 130; 2024: 114; 2025: 106; 2026: 62.

449 papers.

Sort:

By area:

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  2. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  4. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  5. 2022Diffusion Models in Vision: A Surveycited by 1,015
  6. 2022On Distillation of Guided Diffusion Modelscited by 725
  7. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  8. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  9. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 395
  10. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  11. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  12. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
38 more
  1. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  2. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 223
  3. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  4. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  5. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 181
  6. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstructioncited by 180
  7. 2022Lossy Image Compression with Conditional Diffusion Modelscited by 170
  8. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  9. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 130
  10. 2022GENIE: Higher-Order Denoising Diffusion Solverscited by 127
  11. 2022Uncovering the Disentanglement Capability in Text-to-Image Diffusion Modelscited by 116
  12. 2023DCFace: Synthetic Face Generation with Dual Condition Diffusion Modelcited by 116
  13. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latentscited by 104
  14. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 102
  15. 2022Accelerating Diffusion Models via Early Stop of the Diffusion Processcited by 100
  16. 2023Return of Unconditional Generation: A Self-supervised Representation Generation Methodcited by 98
  17. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 98
  18. 2023Diffusion Model as Representation Learnercited by 95
  19. 2023Object-Centric Slot Diffusioncited by 90
  20. 2025Latent Diffusion Model without Variational Autoencodercited by 88
  21. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  22. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  23. 2022DiffFace: Diffusion-based Face Swapping with Facial Guidancecited by 77
  24. 2023SODA: Bottleneck Diffusion Models for Representation Learningcited by 77
  25. 2023PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalizationcited by 77
  26. 2023DiffusionRig: Learning Personalized Priors for Facial Appearance Editingcited by 75
  27. 2023Diffusion Action Segmentationcited by 74
  28. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  29. 2023Benchmarking Autoregressive Conditional Diffusion Models for Turbulent Flow Simulationcited by 71
  30. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
  31. 2022Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidancecited by 67
  32. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 67
  33. 2023DiffusionLight: Light Probes for Free by Painting a Chrome Ballcited by 64
  34. 2023Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Modelscited by 60
  35. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  36. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Modelcited by 58
  37. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 56
  38. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselvescited by 56

50 most cited of 449 All 449 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Transformers with Representation Autoencoders
  2. 2025DGAE: Diffusion-Guided Autoencoder for Efficient Latent Representation Learning
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Models
  4. 2022Semantic Image Synthesis via Diffusion Models
  5. 2023Efficient Transfer Learning in Diffusion Models via Adversarial Noise
  6. 2024Latent Denoising Diffusion GAN: Faster sampling, Higher image quality
  7. 2023Diffusion Models Beat GANs on Image Classification
  8. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesis