mlsanityData to 7 Oct 2026

Estimating High Order Gradients of the Data Distribution by Denoising

arXiv 2111.04726 ↗First posted 8 Nov 2021cs.LG stat.MLcited by 63

Not on the map. Why?

Cited by 44 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The first order derivative of a data density can be estimated efficiently by denoising score matching, and has become an important component in many applications, such as image generation and audio synthesis. Higher order derivatives provide additional local information about the data distribution and enable new applications. Although they can be estimated via automatic differentiation of a learned density model, this can amplify estimation errors and is expensive in high dimensional settings. To overcome these limitations, we propose a method to directly estimate high order derivatives (scores) of a data density from samples. We first show that denoising score matching can be interpreted as a particular case of Tweedie's formula. By leveraging Tweedie's formula on higher order moments, we generalize denoising score matching to estimate higher order derivatives. We demonstrate empirically that models trained with the proposed method can approximate second order derivatives more efficiently and accurately than via automatic differentiation. We show that our models can be used to quantify uncertainty in denoising and to improve the mixing speed of Langevin dynamics via Ozaki discretization for sampling synthetic data and natural images.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 8 Nov 2021.

Builds on

16 of 32 references.

References · 16

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  5. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  7. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  8. 2021How to Train Your Energy-Based Modelscited by 271
  9. 2017User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradientcited by 116
  10. 2020Efficient Learning of Generative Models via Finite-Difference Score Matchingcited by 49
  11. 2019Neural Empirical Bayescited by 46
  12. 2017A-NICE-MC: Adversarial Training for MCMCcited by 36
4 more
  1. 2019High-Order Langevin Diffusion Yields an Accelerated MCMC Algorithmcited by 36
  2. 2018Deep Energy Estimator Networkscited by 34
  3. 2020Nonparametric Score Estimatorscited by 22
  4. 2020A Wasserstein Minimum Velocity Approach to Learning Unnormalized Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 8; 2024: 14; 2025: 16; 2026: 16.

63 papers.

Sort:

By area:

  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  2. 2022Dual Diffusion Implicit Bridges for Image-to-Image Translationcited by 212
  3. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 144
  4. 2022GENIE: Higher-Order Denoising Diffusion Solverscited by 127
  5. 2022Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matchingcited by 79
  6. 2023Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusioncited by 78
  7. 2024Learning Diffusion Priors from Observations by Expectation Maximizationcited by 60
  8. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  9. 2023Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistentcited by 58
  10. 2024Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariancecited by 42
  11. 2022FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equationcited by 36
  12. 2024Variational Diffusion Posterior Sampling with Midpoint Guidancecited by 32
38 more
  1. 2023On the Posterior Distribution in Denoising: Application to Uncertainty Quantificationcited by 31
  2. 2022Learning Physics-Informed Neural Networks without Stacked Back-propagationcited by 23
  3. 2024Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapescited by 20
  4. 2024Diffusion Models as Cartoonists: The Curious Case of High Density Regionscited by 19
  5. 2025Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulationcited by 19
  6. 2021Heavy-tailed denoising score matchingcited by 17
  7. 2023Uncertainty Quantification via Neural Posterior Principal Componentscited by 16
  8. 2024Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costscited by 14
  9. 2024Improving Probabilistic Diffusion Models With Optimal Diagonal Covariance Matchingcited by 13
  10. 2024Diffusion Models in 3D Vision: A Surveycited by 11
  11. 2026Improving Flow Matching by Aligning Flow Divergencecited by 9
  12. 2023Moment Matching Denoising Gibbs Samplingcited by 7
  13. 2024Identifiability Guarantees for Causal Disentanglement from Purely Observational Datacited by 7
  14. 2025Graph-Aware Diffusion for Signal Generationcited by 7
  15. 2026High-accuracy and dimension-free sampling with diffusionscited by 7
  16. 2022Denoising Deep Generative Modelscited by 6
  17. 2023Fit Like You Sample: Sample-Efficient Generalized Score Matching from Fast Mixing Diffusionscited by 6
  18. 2025The Spacetime of Diffusion Models: An Information Geometry Perspectivecited by 6
  19. 2024DOF: Accelerating High-order Differential Operators with Forward Propagationcited by 5
  20. 2024DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Modelscited by 5
  21. 2025Efficient Zero-Shot Inpainting with Decoupled Diffusion Guidancecited by 5
  22. 2024Hierarchical Uncertainty Exploration via Feedforward Posterior Treescited by 4
  23. 2024Moderating the Generalization of Score-based Generative Modelcited by 4
  24. 2025Predicting partially observable dynamical systems via diffusion models with a multiscale inference schemecited by 4
  25. 2023Uncertainty Visualization via Low-Dimensional Posterior Projectionscited by 3
  26. 2024Space-Time Diffusion Bridgecited by 3
  27. 2026Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correctioncited by 3
  28. 2025Benchmarking Diffusion Annealing-Based Bayesian Inverse Problem Solverscited by 2
  29. 2025Score-Based Turbo Message Passing for Plug-and-Play Compressive Image Recoverycited by 2
  30. 2025Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Modelscited by 2
  31. 2026Low-Dimensional Adaptation of Rectified Flow: A Diffusion and Stochastic Localization Perspectivecited by 2
  32. 2026Few-Step Boltzmann Generators via Scalable Likelihood Flow Mapscited by 2
  33. 2025Quantifying the Ease of Reproducing Training Data in Unconditional Diffusion Modelscited by 1
  34. 2025Nonlinear Multiple Response Regression and Learning of Latent Spacescited by 1
  35. 2025Heteroscedasticity of Denoising Score Matching with Generalised Smooth Noisecited by 1
  36. 2026Score-based Metropolis-Hastings for Fractional Langevin Algorithmscited by 1
  37. 2026Outlier-robust Diffusion Posterior Sampling for Bayesian Inverse Problemscited by 1
  38. 2026Multi-Domain Empirical Bayes for Linearly-Mixed Causal Representationscited by 1

50 most cited of 63 All 63 (JSON)