mlsanityData to 14 Oct 2026

DiffEnc: Variational Diffusion with a Learned Encoder

arXiv 2310.19789 ↗First posted 30 Oct 2023cs.LG cs.CV stat.MLcited by 17

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models may be viewed as hierarchical variational autoencoders (VAEs) with two improvements: parameter sharing for the conditional distributions in the generative process and efficient computation of the loss as independent terms over the hierarchy. We consider two changes to the diffusion model that retain these advantages while adding flexibility to the model. Firstly, we introduce a data- and depth-dependent mean function in the diffusion process, which leads to a modified diffusion loss. Our proposed framework, DiffEnc, achieves a statistically significant improvement in likelihood on CIFAR-10. Secondly, we let the ratio of the noise variance of the reverse encoder process and the generative process be a free weight parameter rather than being fixed to 1. This leads to theoretical insights: For a finite depth hierarchy, the evidence lower bound (ELBO) can be used as an objective for a weighted diffusion loss approach and for optimizing the noise schedule specifically for inference. For the infinite-depth hierarchy, on the other hand, the weight parameter has to be 1 to have a well-defined ELBO.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 53 of its 61 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

53 of 61 references.

Closest ancestors · 26

Same direction, newest first

  1. 2023Neural Diffusion Models12 Oct
  2. 2023Common Diffusion Noise Schedules and Sample Steps are Flawed15 May
  3. 2023Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs6 May
  4. 2023Reflected Diffusion Models10 Apr
  5. 2023TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation7 Mar
  6. 2023Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusions14 Feb
  7. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  8. 2022Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models28 Nov
  9. 2022Blurring Diffusion Models12 Sep
  10. 2022Soft Diffusion: Score Matching for General Corruptions12 Sep
  11. 2022Entropy-driven Sampling and Training Scheme for Conditional Diffusion Generation23 Jun
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
14 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  2. 2022Maximum Likelihood Training of Implicit Nonlinear Diffusion Models27 May
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  4. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representation30 Nov
  5. 2021Variational Diffusion Models1 Jul
  6. 2021Score-based Generative Modeling in Latent Space10 Jun
  7. 2021Learning to Efficiently Sample from Diffusion Probabilistic Models7 Jun
  8. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  9. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  10. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  11. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  13. 2020Denoising Diffusion Implicit Models6 Oct
  14. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 27

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  8. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  9. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  10. 2016Pixel Recurrent Neural Networkscited by 816
  11. 2018Image Transformercited by 772
  12. 2023Extracting Training Data from Diffusion Modelscited by 732
15 more
  1. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  2. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  3. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  4. 2016Ladder Variational Autoencoderscited by 318
  5. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  6. 2015A note on the evaluation of generative modelscited by 292
  7. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 264
  8. 2021Autoregressive Diffusion Modelscited by 210
  9. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  10. 2022Generative Modelling With Inverse Heat Dissipationcited by 167
  11. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speechcited by 157
  12. 2022Diffusion Models for Video Prediction and Infillingcited by 135
  13. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusioncited by 78
  14. 2021Diffusion-Based Representation Learningcited by 57
  15. 2021Consistency Regularization for Variational Auto-Encoderscited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 11; 2026: 3.

17 papers.

Sort:

By area:

  1. 2023Diffusion Models With Learned Adaptive Noisecited by 51
  2. 2024Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modellingcited by 45
  3. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  4. 2025SDE Matching: Scalable and Simulation-Free Training of Latent Stochastic Differential Equationscited by 23
  5. 2024Diffusion Model Guided Sampling with Pixel-Wise Aleatoric Uncertainty Estimationcited by 20
  6. 2025VCT: Training Consistency Models with Variational Noise Couplingcited by 15
  7. 2025Equivariant Neural Diffusion for Molecule Generationcited by 13
  8. 2025Adaptive Destruction Processes for Diffusion Samplerscited by 6
  9. 2025Training Latent Diffusion Models with Interacting Particle Algorithmscited by 4
  10. 2025CoVAE: Consistency Training of Variational Autoencoderscited by 3
  11. 2026JEDI: Joint Embedding Diffusion World Model for Online Model-Based Reinforcement Learningcited by 2
  12. 2025On Designing Diffusion Autoencoders for Efficient Generation and Representation Learningcited by 1
5 more
  1. 2025RestoreGrad: Signal Restoration Using Conditional Denoising Diffusion Models with Jointly Learned Priorcited by 0
  2. 2025Performance Asymmetry in Model-Based Reinforcement Learningcited by 0
  3. 2025Breaking the Likelihood-Quality Trade-off in Diffusion Models by Merging Pretrained Expertscited by 0
  4. 2026Introduction to Stochastic Differential Equations for Generative Machine Learning: A Variational Perspectivecited by 0
  5. 2026Gaussian Flow Dynamics: Simulation-Free Neural SDE Learning Beyond One-Time Marginalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Understanding Diffusion Models: A Unified Perspective
  2. 2021Diffusion Priors In Variational Autoencoders
  3. 2023Variational Diffusion Auto-encoder: Latent Space Extraction from Pre-trained Diffusion Models
  4. 2026The Diffusion Encoder
  5. 2025Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis
  6. 2025Diffusion Models: A Mathematical Introduction
  7. 2025Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds
  8. 2024Multimodal ELBO with Diffusion Decoders