# Variance Reduction for Expectations with Diffusion Teachers

arXiv: https://arxiv.org/abs/2605.21489 · HTML: https://mlsanity.com/paper/2605.21489/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 38 of its 102 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 38)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [Solving Inverse Problems in Medical Imaging with Score-Based Generative Models](https://mlsanity.com/paper/2111.08005/) (2021)
- [Improved Techniques for Training Consistency Models](https://mlsanity.com/paper/2310.14189/) (2023)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [DreamGaussian4D: Generative 4D Gaussian Splatting](https://mlsanity.com/paper/2312.17142/) (2023)
- [Text-to-3D with Classifier Score Distillation](https://mlsanity.com/paper/2310.19415/) (2023)
- [DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models](https://mlsanity.com/paper/2310.00902/) (2023)
- [HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance](https://mlsanity.com/paper/2305.18766/) (2023)
- [Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation](https://mlsanity.com/paper/2106.05527/) (2021)
- [LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models](https://mlsanity.com/paper/2411.09595/) (2024)
- [Align Your Flow: Scaling Continuous-Time Flow Map Distillation](https://mlsanity.com/paper/2506.14603/) (2025)
- [Multistep Distillation of Diffusion Models via Moment Matching](https://mlsanity.com/paper/2406.04103/) (2024)
- [Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion](https://mlsanity.com/paper/2406.04338/) (2024)
- [Progressive Distillation for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2202.00512/) (2022)
- [EM Distillation for One-step Diffusion Models](https://mlsanity.com/paper/2405.16852/) (2024)
- [VidGen-1M: A Large-Scale Dataset for Text-to-video Generation](https://mlsanity.com/paper/2408.02629/) (2024)
- [The Journey, Not the Destination: How Data Guides Diffusion Models](https://mlsanity.com/paper/2312.06205/) (2023)
- [Rethinking Score Distillation as a Bridge Between Image Distributions](https://mlsanity.com/paper/2406.09417/) (2024)
- [Training Data Attribution via Approximate Unrolled Differentiation](https://mlsanity.com/paper/2405.12186/) (2024)

## Built on by (most cited first, 2 of 2)

- [Addressable Memory for Video World Models](https://mlsanity.com/paper/2608.07408/) (2026)
- [Variance Reduction on the Camera Axis: Multi-View Score Distillation for 3D](https://mlsanity.com/paper/2606.29964/) (2026)

## Abstract (arXiv)

Pretrained diffusion models serve as frozen teachers feeding downstream pipelines such as text-to-3D, single-step distillation, and data attribution. The teacher gradients these pipelines consume are Monte Carlo (MC) expectations over noise levels and Gaussian noise samples; their estimator variance dominates compute cost because each draw requires expensive upstream work (rendering, simulation, encoding). We introduce CARV, a compute-aware variance-accounting framework that motivates a hierarchical MC estimator: amortize the expensive upstream computation over cheap diffusion-noise resamples, sharpened by timestep importance sampling and a stratified-inverse-CDF construction. In our text-to-3D distillation and attribution experiments, CARV delivers 2-3x effective compute multipliers (most from amortized reuse; ~25% additional from IS+stratification) without changing the objective; in single-step distillation, the same techniques cut gradient variance by an order of magnitude but do not improve downstream FID, marking the regime where MC variance is no longer the bottleneck.