mlsanityData to 7 Oct 2026

Maximum Likelihood Training of Score-Based Diffusion Models

arXiv 2101.09258 ↗First posted 22 Jan 2021stat.ML cs.LGcited by 670

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32x32 without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.

Key facts. Cited by 670 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 35 of its 65 references that are in the map. First posted on arXiv 22 Jan 2021.

Builds on

35 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  2. 2020Denoising Diffusion Probabilistic Models19 Jun
  3. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 32

  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2016Density estimation using Real NVPcited by 2,029
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  8. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  9. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  10. 2016Pixel Recurrent Neural Networkscited by 814
  11. 2018Image Transformercited by 770
  12. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
20 more
  1. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  2. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  3. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  4. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  5. 2015A note on the evaluation of generative modelscited by 291
  6. 2020Scalable Gradients for Stochastic Differential Equationscited by 264
  7. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  8. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 258
  9. 2020Learning Gradient Fields for Shape Generationcited by 245
  10. 2017PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examplescited by 243
  11. 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 168
  12. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  13. 2020Interacting particle solutions of Fokker-Planck equations through gradient-log-density estimationcited by 87
  14. 2019Preventing Posterior Collapse with delta-VAEscited by 86
  15. 2017Good Semi-supervised Learning that Requires a Bad GANcited by 85
  16. 2019Practical Lossless Compression with Latent Variables using Bits Back Codingcited by 85
  17. 2019Integer Discrete Flows and Lossless Compressioncited by 82
  18. 2013RNADE: The real-valued neural autoregressive density-estimatorcited by 65
  19. 2020VFlow: More Expressive Generative Flows with Variational Data Augmentationcited by 38
  20. 2019Compression with Flows via Local Bits-Back Codingcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 79; 2023: 161; 2024: 171; 2025: 158; 2026: 85.

670 papers.

Sort:

By area:

  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  4. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  5. 2021Variational Diffusion Modelscited by 1,145
  6. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  7. 2022Diffusion Models in Vision: A Surveycited by 1,008
  8. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  9. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  10. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  11. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  12. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
38 more
  1. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2021Score-based Generative Modeling in Latent Spacecited by 634
  4. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  5. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 526
  6. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  7. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  8. 2022Diffusion Models for Adversarial Purificationcited by 469
  9. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 448
  10. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptionscited by 440
  11. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  12. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  13. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  14. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 331
  15. 2021Score-based diffusion models for accelerated MRIcited by 308
  16. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  17. 2022Novel View Synthesis with Diffusion Modelscited by 299
  18. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 293
  19. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  20. 2023Multisample Flow Matching: Straightening Flows with Minibatch Couplingscited by 267
  21. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  22. 2022A Survey on Generative Diffusion Modelcited by 238
  23. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  24. 2022Convergence of denoising diffusion models under the manifold hypothesiscited by 237
  25. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusioncited by 230
  26. 2023Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localizationcited by 230
  27. 2023A Variational Perspective on Solving Inverse Problems with Diffusion Modelscited by 217
  28. 2021Likelihood Training of Schr"odinger Bridge using Forward-Backward SDEs Theorycited by 216
  29. 2022Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptionscited by 215
  30. 2023I2SB: Image-to-Image Schr"odinger Bridgecited by 214
  31. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusioncited by 204
  32. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 204
  33. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  34. 2023A Survey on Video Diffusion Modelscited by 203
  35. 2022Continuous diffusion for categorical datacited by 202
  36. 2021Conditional Image Generation with Score-Based Diffusion Modelscited by 195
  37. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  38. 2023Image Restoration with Mean-Reverting Stochastic Differential Equationscited by 182

50 most cited of 670 All 670 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Score Augmentation for Diffusion Models
  2. 2024Efficient Denoising using Score Embedding in Score-based Diffusion Models
  3. 2026Generalization Properties of Score-matching Diffusion Models for Intrinsically Low-dimensional Data
  4. 2024Random Walks with Tweedie: A Unified View of Score-Based Diffusion Models
  5. 2025Dimension-free Score Matching and Time Bootstrapping for Diffusion Models
  6. 2026From Score Approximation to Distribution Approximation in Score-Based Diffusion Models
  7. 2023Stable Target Field for Reduced Variance Score Estimation in Diffusion Models
  8. 2024O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions