mlsanityData to 7 Oct 2026

Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching

arXiv 2206.08265 ↗First posted 16 Jun 2022stat.ML cs.LGcited by 79

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Score-based generative models have excellent performance in terms of generation quality and likelihood. They model the data distribution by matching a parameterized score network with first-order data score functions. The score network can be used to define an ODE ("score-based diffusion ODE") for exact likelihood evaluation. However, the relationship between the likelihood of the ODE and the score matching objective is unclear. In this work, we prove that matching the first-order score is not sufficient to maximize the likelihood of the ODE, by showing a gap between the maximum likelihood and score matching objectives. To fill up this gap, we show that the negative likelihood of the ODE can be bounded by controlling the first, second, and third-order score matching errors; and we further present a novel high-order denoising score matching method to enable maximum likelihood training of score-based diffusion ODEs. Our algorithm guarantees that the higher-order matching error is bounded by the training error and the lower-order errors. We empirically observe that by high-order score matching, score-based diffusion ODEs achieve better likelihood on both synthetic data and CIFAR-10, while retaining the high generation quality.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 21 of its 35 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

21 of 35 references.

Closest ancestors · 10

Same direction, newest first

  1. 2022Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models15 Jun
  2. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusion14 Dec
  3. 2021Variational Diffusion Models1 Jul
  4. 2021Score-based Generative Modeling in Latent Space10 Jun
  5. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matching5 Jun
  6. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  7. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  9. 2020Denoising Diffusion Probabilistic Models19 Jun
  10. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 11

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  5. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  6. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  7. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  8. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  9. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  10. 2020How to train your neural ODE: the world of Jacobian and kinetic regularizationcited by 228
  11. 2021Estimating High Order Gradients of the Data Distribution by Denoisingcited by 63

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 21; 2024: 29; 2025: 15; 2026: 9.

79 papers.

Sort:

By area:

  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  2. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  3. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  4. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  5. 2023Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusioncited by 448
  6. 2022A Survey on Generative Diffusion Modelcited by 238
  7. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 204
  8. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  9. 2023DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statisticscited by 161
  10. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  11. 2023Score-Based Diffusion Models as Principled Priors for Inverse Imagingcited by 126
  12. 2025Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistencycited by 81
38 more
  1. 2023The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editingcited by 61
  2. 2023Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEscited by 54
  3. 2023Diffusion Models With Learned Adaptive Noisecited by 50
  4. 2024Diffusion Bridge Implicit Modelscited by 47
  5. 2024Scalable Diffusion Models with State Space Backbonecited by 45
  6. 2024Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariancecited by 42
  7. 2022FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equationcited by 36
  8. 2024Stochastic Runge-Kutta Methods: Provable Acceleration of Diffusion Modelscited by 35
  9. 2024DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidancecited by 33
  10. 2024Improving Diffusion-Based Image Synthesis with Context Predictioncited by 32
  11. 2023On the Posterior Distribution in Denoising: Application to Uncertainty Quantificationcited by 31
  12. 2024Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equationscited by 28
  13. 2023Towards Predicting Equilibrium Distributions for Molecular Systems with Deep Learningcited by 26
  14. 2024Training Unbiased Diffusion Models From Biased Datasetcited by 25
  15. 2024Your Diffusion Model is Secretly a Certifiably Robust Classifiercited by 24
  16. 2026Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Modelscited by 23
  17. 2024Consistency Diffusion Bridge Modelscited by 22
  18. 2025Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminatorcited by 22
  19. 2023Lipschitz Singularities in Diffusion Modelscited by 21
  20. 2024Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapescited by 20
  21. 2024Diffusion Models as Cartoonists: The Curious Case of High Density Regionscited by 19
  22. 2025Towards Hierarchical Rectified Flowcited by 19
  23. 2023Gaussian Interpolation Flowscited by 17
  24. 2023AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computationcited by 14
  25. 2024Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costscited by 14
  26. 2023Entropy-based Training Methods for Scalable Neural Implicit Samplercited by 13
  27. 2024Understanding Diffusion Models by Feynman's Path Integralcited by 13
  28. 2023High-dimensional and Permutation Invariant Anomaly Detectioncited by 12
  29. 2023Closing the ODE-SDE gap in score-based diffusion models through the Fokker-Planck equationcited by 12
  30. 2024Structure-Guided Adversarial Training of Diffusion Modelscited by 11
  31. 2024Eta Inversion: Designing an Optimal Eta Function for Diffusion-based Real Image Editingcited by 11
  32. 2024Neural Residual Diffusion Models for Deep Scalable Vision Generationcited by 9
  33. 2026Improving Flow Matching by Aligning Flow Divergencecited by 9
  34. 2025Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flowscited by 8
  35. 2024Improving Adversarial Energy-Based Model via Diffusion Processcited by 7
  36. 2023Preconditioned Score-based Generative Modelscited by 6
  37. 2025The Spacetime of Diffusion Models: An Information Geometry Perspectivecited by 6
  38. 2023Convergence guarantee for consistency modelscited by 5

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024What's the score? Automated Denoising Score Matching for Nonlinear Diffusions
  2. 2022From Denoising Diffusions to Denoising Markov Models
  3. 2022How Much is Enough? A Study on Diffusion Times in Score-based Generative Models
  4. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions
  5. 2026Where the Score Lives: A Wavelet View of Diffusion
  6. 2025Minimax Optimality of the Probability Flow ODE for Diffusion Models
  7. 2024Nearest Neighbour Score Estimators for Diffusion Generative Models
  8. 2024Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization