mlsanityData to 7 Oct 2026

Variance reduction of diffusion model's gradients with Taylor approximation-based control variate

arXiv 2408.12270 ↗First posted 22 Aug 2024cs.LG cs.AIcited by 6

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Score-based models, trained with denoising score matching, are remarkably effective in generating high dimensional data. However, the high variance of their training objective hinders optimisation. We attempt to reduce it with a control variate, derived via a k-th order Taylor expansion on the training objective and its gradient. We prove an equivalence between the two and demonstrate empirically the effectiveness of our approach on a low dimensional problem setting; and study its effect on larger problems.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 24 of its 41 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

24 of 41 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  2. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matching5 Jun
  3. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  5. 2020Denoising Diffusion Probabilistic Models19 Jun
  6. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 18

  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  3. 2016Variational Inference: A Review for Statisticianscited by 1,466
  4. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  5. 2019An Introduction to Variational Autoencoderscited by 997
  6. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  7. 2013Black Box Variational Inferencecited by 317
  8. 2021How to Train Your Energy-Based Modelscited by 271
  9. 2016Benefits of depth in neural networkscited by 248
  10. 2017Backpropagation through the Void: Optimizing control variates for black-box gradient estimationcited by 104
  11. 2015Representation Benefits of Deep Feedforward Networkscited by 84
  12. 2019BackPACK: Packing more into backpropcited by 77
6 more
  1. 2018Tighter Variational Bounds are Not Necessarily Bettercited by 76
  2. 2022A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Modelcited by 44
  3. 2017Reducing Reparameterization Gradient Variancecited by 27
  4. 2018Using Large Ensembles of Control Variates for Variational Inferencecited by 21
  5. 2020A Wasserstein Minimum Velocity Approach to Learning Unnormalized Modelscited by 9
  6. 2020Amortized variance reduction for doubly stochastic objectivescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2024Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellencycited by 21
  2. 2025Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlocited by 18
  3. 2026Stable Velocity: A Variance Perspective on Flow Matchingcited by 10
  4. 2025Modular MeanFlow: Towards Stable and Scalable One-Step Generative Modelingcited by 8
  5. 2025Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Modelscited by 2
  6. 2025D2 Actor Critic: Diffusion Actor Meets Distributional Criticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Control Variate Score Matching for Diffusion Models
  2. 2025Faster Diffusion Models via Higher-Order Approximation
  3. 2026Variance-Reduced Diffusion Sampling via Target Score Identity
  4. 2023Stable Target Field for Reduced Variance Score Estimation in Diffusion Models
  5. 2024Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models
  6. 2025Optimization-Free Diffusion Model -- A Perturbation Theory Approach
  7. 2023To smooth a cloud or to pin it down: Guarantees and Insights on Score Matching in Denoising Diffusion Models
  8. 2022Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching