# Estimating High Order Gradients of the Data Distribution by Denoising

arXiv: https://arxiv.org/abs/2111.04726 · HTML: https://mlsanity.com/paper/2111.04726/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 8 Nov 2021.

## Builds on (most cited first, 16 of 16)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient](https://mlsanity.com/paper/1710.00095/) (2017)
- [Efficient Learning of Generative Models via Finite-Difference Score Matching](https://mlsanity.com/paper/2007.03317/) (2020)
- [Neural Empirical Bayes](https://mlsanity.com/paper/1903.02334/) (2019)
- [A-NICE-MC: Adversarial Training for MCMC](https://mlsanity.com/paper/1706.07561/) (2017)
- [High-Order Langevin Diffusion Yields an Accelerated MCMC Algorithm](https://mlsanity.com/paper/1908.10859/) (2019)
- [Deep Energy Estimator Networks](https://mlsanity.com/paper/1805.08306/) (2018)
- [Nonparametric Score Estimators](https://mlsanity.com/paper/2005.10099/) (2020)
- [A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models](https://mlsanity.com/paper/2002.07501/) (2020)

## Built on by (most cited first, 30 of 64)

- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Dual Diffusion Implicit Bridges for Image-to-Image Translation](https://mlsanity.com/paper/2203.08382/) (2022)
- [Concrete Score Matching: Generalized Score Matching for Discrete Data](https://mlsanity.com/paper/2211.00802/) (2022)
- [GENIE: Higher-Order Denoising Diffusion Solvers](https://mlsanity.com/paper/2210.05475/) (2022)
- [Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching](https://mlsanity.com/paper/2206.08265/) (2022)
- [Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion](https://mlsanity.com/paper/2312.00852/) (2023)
- [Learning Diffusion Priors from Observations by Expectation Maximization](https://mlsanity.com/paper/2405.13712/) (2024)
- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)
- [Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent](https://mlsanity.com/paper/2302.09057/) (2023)
- [Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance](https://mlsanity.com/paper/2402.02149/) (2024)
- [FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equation](https://mlsanity.com/paper/2210.04296/) (2022)
- [On the Posterior Distribution in Denoising: Application to Uncertainty Quantification](https://mlsanity.com/paper/2309.13598/) (2023)
- [Variational Diffusion Posterior Sampling with Midpoint Guidance](https://mlsanity.com/paper/2410.09945/) (2024)
- [Learning Physics-Informed Neural Networks without Stacked Back-propagation](https://mlsanity.com/paper/2202.09340/) (2022)
- [Diffusion Models as Cartoonists: The Curious Case of High Density Regions](https://mlsanity.com/paper/2411.01293/) (2024)
- [Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapes](https://mlsanity.com/paper/2412.04140/) (2024)
- [Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation](https://mlsanity.com/paper/2507.02608/) (2025)
- [Heavy-tailed denoising score matching](https://mlsanity.com/paper/2112.09788/) (2021)
- [Uncertainty Quantification via Neural Posterior Principal Components](https://mlsanity.com/paper/2309.15533/) (2023)
- [Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs](https://mlsanity.com/paper/2410.11149/) (2024)
- [Improving Probabilistic Diffusion Models With Optimal Diagonal Covariance Matching](https://mlsanity.com/paper/2406.10808/) (2024)
- [Diffusion Models in 3D Vision: A Survey](https://mlsanity.com/paper/2410.04738/) (2024)
- [Improving Flow Matching by Aligning Flow Divergence](https://mlsanity.com/paper/2602.00869/) (2026)
- [Graph-Aware Diffusion for Signal Generation](https://mlsanity.com/paper/2510.05036/) (2025)
- [Moment Matching Denoising Gibbs Sampling](https://mlsanity.com/paper/2305.11650/) (2023)
- [Identifiability Guarantees for Causal Disentanglement from Purely Observational Data](https://mlsanity.com/paper/2410.23620/) (2024)
- [High-accuracy and dimension-free sampling with diffusions](https://mlsanity.com/paper/2601.10708/) (2026)
- [Denoising Deep Generative Models](https://mlsanity.com/paper/2212.01265/) (2022)
- [Fit Like You Sample: Sample-Efficient Generalized Score Matching from Fast Mixing Diffusions](https://mlsanity.com/paper/2306.09332/) (2023)
- [The Spacetime of Diffusion Models: An Information Geometry Perspective](https://mlsanity.com/paper/2505.17517/) (2025)

## Abstract (arXiv)

The first order derivative of a data density can be estimated efficiently by denoising score matching, and has become an important component in many applications, such as image generation and audio synthesis. Higher order derivatives provide additional local information about the data distribution and enable new applications. Although they can be estimated via automatic differentiation of a learned density model, this can amplify estimation errors and is expensive in high dimensional settings. To overcome these limitations, we propose a method to directly estimate high order derivatives (scores) of a data density from samples. We first show that denoising score matching can be interpreted as a particular case of Tweedie's formula. By leveraging Tweedie's formula on higher order moments, we generalize denoising score matching to estimate higher order derivatives. We demonstrate empirically that models trained with the proposed method can approximate second order derivatives more efficiently and accurately than via automatic differentiation. We show that our models can be used to quantify uncertainty in denoising and to improve the mixing speed of Langevin dynamics via Ozaki discretization for sampling synthetic data and natural images.