# Heavy-tailed denoising score matching arXiv: https://arxiv.org/abs/2112.09788 · HTML: https://mlsanity.com/paper/2112.09788/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. - Builds on 26 of its 53 references that are in the map. - First posted on arXiv 17 Dec 2021. ## Placed in - Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/) ## Builds on (most cited first, 26 of 26) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020) - [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018) - [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014) - [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019) - [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016) - [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015) - [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018) - [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021) - [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020) - [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017) - [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021) - [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021) - [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021) - [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020) - [Assessing Generative Models via Precision and Recall](https://mlsanity.com/paper/1806.00035/) (2018) - [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019) - [Gotta Go Fast When Generating Data with Score-Based Models](https://mlsanity.com/paper/2105.14080/) (2021) - [A Variational Perspective on Diffusion-Based Generative Models and Score Matching](https://mlsanity.com/paper/2106.02808/) (2021) - [Sharp convergence rates for Langevin dynamics in the nonconvex setting](https://mlsanity.com/paper/1805.01648/) (2018) - [Estimating High Order Gradients of the Data Distribution by Denoising](https://mlsanity.com/paper/2111.04726/) (2021) - [Non Gaussian Denoising Diffusion Models](https://mlsanity.com/paper/2106.07582/) (2021) - [Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning](https://mlsanity.com/paper/1907.00865/) (2019) - [High-Order Langevin Diffusion Yields an Accelerated MCMC Algorithm](https://mlsanity.com/paper/1908.10859/) (2019) - [Fractional Langevin Monte Carlo: Exploring L'evy Driven Stochastic Differential Equations for Markov Chain Monte Carlo](https://mlsanity.com/paper/1706.03649/) (2017) ## Built on by (most cited first, 18 of 18) - [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022) - [Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration](https://mlsanity.com/paper/2303.11435/) (2023) - [Soft Diffusion: Score Matching for General Corruptions](https://mlsanity.com/paper/2209.05442/) (2022) - [Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent](https://mlsanity.com/paper/2302.09057/) (2023) - [Iterative α-(de)Blending: a Minimalist Deterministic Diffusion Model](https://mlsanity.com/paper/2305.03486/) (2023) - [Sampling, Diffusions, and Stochastic Localization](https://mlsanity.com/paper/2305.10690/) (2023) - [Stage-by-stage Wavelet Optimization Refinement Diffusion Model for Sparse-View CT Reconstruction](https://mlsanity.com/paper/2308.15942/) (2023) - [DiracDiffusion: Denoising and Incremental Reconstruction with Assured Data-Consistency](https://mlsanity.com/paper/2303.14353/) (2023) - [A solvable model of learning generative diffusion: theory and insights](https://mlsanity.com/paper/2501.03937/) (2025) - [Diffusion models with location-scale noise](https://mlsanity.com/paper/2304.05907/) (2023) - [Generative modelling with jump-diffusions](https://mlsanity.com/paper/2503.06558/) (2025) - [Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models](https://mlsanity.com/paper/2506.03849/) (2025) - [Diffusion Models with Heavy-Tailed Targets: Score Estimation and Sampling Guarantees](https://mlsanity.com/paper/2601.06715/) (2026) - [A noise-corrected Langevin algorithm and sampling by half-denoising](https://mlsanity.com/paper/2410.05837/) (2024) - [Offline Reinforcement Learning with Penalized Action Noise Injection](https://mlsanity.com/paper/2507.02356/) (2025) - [Heteroscedasticity of Denoising Score Matching with Generalised Smooth Noise](https://mlsanity.com/paper/2508.01597/) (2025) - [Ordered-subsets Multi-diffusion Model for Sparse-view CT Reconstruction](https://mlsanity.com/paper/2505.09985/) (2025) - [From Global to Tail Convergence: Dynamics of α-Stable Score-Based Generative Models](https://mlsanity.com/paper/2610.19773/) (2026) ## Abstract (arXiv) Score-based model research in the last few years has produced state of the art generative models by employing Gaussian denoising score-matching (DSM). However, the Gaussian noise assumption has several high-dimensional limitations, motivating a more concrete route toward even higher dimension PDF estimation in future. We outline this limitation, before extending the theory to a broader family of noising distributions -- namely, the generalised normal distribution. To theoretically ground this, we relax a key assumption in (denoising) score matching theory, demonstrating that distributions which are differentiable almost everywhere permit the same objective simplification as Gaussians. For noise vector norm distributions, we demonstrate favourable concentration of measure in the high-dimensional spaces prevalent in deep learning. In the process, we uncover a skewed noise vector norm distribution and develop an iterative noise scaling algorithm to consistently initialise the multiple levels of noise in annealed Langevin dynamics (LD). On the practical side, our use of heavy-tailed DSM leads to improved score estimation, controllable sampling convergence, and more balanced unconditional generative performance for imbalanced datasets.