mlsanityData to 7 Oct 2026

Score-based Generative Modeling in Latent Space

arXiv 2106.05931 ↗First posted 10 Jun 2021stat.ML cs.LGcited by 634

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution coverage. However, they are usually applied directly in data space and often require thousands of network evaluations for sampling. Here, we propose the Latent Score-based Generative Model (LSGM), a novel approach that trains SGMs in a latent space, relying on the variational autoencoder framework. Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling. To enable training LSGMs end-to-end in a scalable and stable manner, we (i) introduce a new score-matching objective suitable to the LSGM setting, (ii) propose a novel parameterization of the score function that allows SGM to focus on the mismatch of the target distribution with respect to a simple Normal one, and (iii) analytically derive multiple techniques for variance reduction of the training objective. LSGM obtains a state-of-the-art FID score of 2.10 on CIFAR-10, outperforming all existing generative results on this dataset. On CelebA-HQ-256, LSGM is on a par with previous SGMs in sample quality while outperforming them in sampling time by two orders of magnitude. In modeling binary images, LSGM achieves state-of-the-art likelihood on the binarized OMNIGLOT dataset. Our project page and code can be found at https://nvlabs.github.io/LSGM .

Key facts. Cited by 634 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 85 of its 110 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

85 of 110 references.

Closest ancestors · 10

Same direction, newest first

  1. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matching5 Jun
  2. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  3. 2021Noise Estimation for Generative Diffusion Models6 Apr
  4. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  5. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  6. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed7 Jan
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  8. 2020Denoising Diffusion Implicit Models6 Oct
  9. 2020Denoising Diffusion Probabilistic Models19 Jun
  10. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 75

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2018Neural Ordinary Differential Equationscited by 3,733
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  9. 2015Variational Inference with Normalizing Flowscited by 2,161
  10. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
38 more
  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  2. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  3. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  4. 2018Junction Tree Variational Autoencoder for Molecular Graph Generationcited by 647
  5. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  6. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  7. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  8. 2015Generating Sentences from a Continuous Spacecited by 607
  9. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  10. 2020Jukebox: A Generative Model for Musiccited by 535
  11. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  12. 2015Importance Weighted Autoencoderscited by 454
  13. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  14. 2019Augmented Neural ODEscited by 396
  15. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  16. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  17. 2018GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoderscited by 375
  18. 2016Ladder Variational Autoencoderscited by 315
  19. 2020Permutation Invariant Graph Generation via Score-Based Generative Modelingcited by 263
  20. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  21. 2017VAE with a VampPriorcited by 247
  22. 2020Learning Gradient Fields for Shape Generationcited by 245
  23. 2019Efficient Graph Generation with Graph Recurrent Attention Networkscited by 231
  24. 2019Diagnosing and Enhancing VAE Modelscited by 192
  25. 2016Variational Lossy Autoencodercited by 175
  26. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speechcited by 156
  27. 2020Adversarial Latent Autoencoderscited by 155
  28. 2021Symbolic Music Generation with Diffusion Modelscited by 152
  29. 2021Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizationcited by 141
  30. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  31. 2019From Variational to Deterministic Autoencoderscited by 131
  32. 2016Discrete Variational Autoencoderscited by 124
  33. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 122
  34. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  35. 2013Generalized Denoising Auto-Encoders as Generative Modelscited by 113
  36. 2018Graphite: Iterative Generative Modeling of Graphscited by 112
  37. 2019BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modelingcited by 109
  38. 2020Learning Latent Space Energy-Based Prior Modelcited by 104

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 95; 2023: 198; 2024: 159; 2025: 114; 2026: 50.

634 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  6. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  7. 2023Consistency Modelscited by 1,778
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  9. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  10. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  11. 2021Variational Diffusion Modelscited by 1,145
  12. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
38 more
  1. 2022Diffusion Models in Vision: A Surveycited by 1,008
  2. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  3. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  4. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  5. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  6. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  7. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  8. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  9. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  10. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  11. 2022Diffusion Models for Adversarial Purificationcited by 469
  12. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  13. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  14. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 448
  15. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptionscited by 440
  16. 2022Blended Latent Diffusioncited by 430
  17. 2023Improved Techniques for Training Consistency Modelscited by 430
  18. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  19. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  20. 2023DeepCache: Accelerating Diffusion Models for Freecited by 346
  21. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 331
  22. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
  23. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  24. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 293
  25. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  26. 20223D Neural Field Generation using Triplane Diffusioncited by 274
  27. 2022Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equationscited by 273
  28. 2022Torsional Diffusion for Molecular Conformer Generationcited by 262
  29. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  30. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  31. 2022A Survey on Generative Diffusion Modelcited by 238
  32. 2022Convergence of denoising diffusion models under the manifold hypothesiscited by 237
  33. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 232
  34. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusioncited by 230
  35. 2022Diffusion models as plug-and-play priorscited by 230
  36. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  37. 2023A Variational Perspective on Solving Inverse Problems with Diffusion Modelscited by 217
  38. 2021Likelihood Training of Schr"odinger Bridge using Forward-Backward SDEs Theorycited by 216

50 most cited of 634 All 634 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Good Score Does not Lead to A Good Generative Model
  2. 2024Moderating the Generalization of Score-based Generative Model
  3. 2024An analysis of the noise schedule for score-based generative models
  4. 2026Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models
  5. 2022Convergence for score-based generative modeling with polynomial complexity
  6. 2024Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective
  7. 2022Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptions
  8. 2023Score-based Conditional Generation with Fewer Labeled Data by Self-calibrating Classifier Guidance