# Variance reduction of diffusion model's gradients with Taylor approximation-based control variate

arXiv: https://arxiv.org/abs/2408.12270 · HTML: https://mlsanity.com/paper/2408.12270/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 24 of 24)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [A Variational Perspective on Diffusion-Based Generative Models and Score Matching](https://mlsanity.com/paper/2106.02808/) (2021)
- [Backpropagation through the Void: Optimizing control variates for black-box gradient estimation](https://mlsanity.com/paper/1711.00123/) (2017)
- [Representation Benefits of Deep Feedforward Networks](https://mlsanity.com/paper/1509.08101/) (2015)
- [BackPACK: Packing more into backprop](https://mlsanity.com/paper/1912.10985/) (2019)
- [Tighter Variational Bounds are Not Necessarily Better](https://mlsanity.com/paper/1802.04537/) (2018)
- [A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model](https://mlsanity.com/paper/2205.06924/) (2022)
- [Reducing Reparameterization Gradient Variance](https://mlsanity.com/paper/1705.07880/) (2017)
- [Using Large Ensembles of Control Variates for Variational Inference](https://mlsanity.com/paper/1810.12482/) (2018)
- [A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models](https://mlsanity.com/paper/2002.07501/) (2020)
- [Amortized variance reduction for doubly stochastic objectives](https://mlsanity.com/paper/2003.04125/) (2020)

## Built on by (most cited first, 6 of 6)

- [Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency](https://mlsanity.com/paper/2410.06025/) (2024)
- [Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo](https://mlsanity.com/paper/2502.12786/) (2025)
- [Stable Velocity: A Variance Perspective on Flow Matching](https://mlsanity.com/paper/2602.05435/) (2026)
- [Modular MeanFlow: Towards Stable and Scalable One-Step Generative Modeling](https://mlsanity.com/paper/2508.17426/) (2025)
- [Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models](https://mlsanity.com/paper/2511.18159/) (2025)
- [D2 Actor Critic: Diffusion Actor Meets Distributional Critic](https://mlsanity.com/paper/2510.03508/) (2025)

## Abstract (arXiv)

Score-based models, trained with denoising score matching, are remarkably effective in generating high dimensional data. However, the high variance of their training objective hinders optimisation. We attempt to reduce it with a control variate, derived via a k-th order Taylor expansion on the training objective and its gradient. We prove an equivalence between the two and demonstrate empirically the effectiveness of our approach on a low dimensional problem setting; and study its effect on larger problems.