# SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity

arXiv: https://arxiv.org/abs/2401.00604 · HTML: https://mlsanity.com/paper/2401.00604/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 61 references that are in the map.
- First posted on arXiv 31 Dec 2023.

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [3D Neural Field Generation using Triplane Diffusion](https://mlsanity.com/paper/2211.16677/) (2022)
- [Learning Gradient Fields for Shape Generation](https://mlsanity.com/paper/2008.06520/) (2020)

## Built on by (most cited first, 23 of 23)

- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation](https://mlsanity.com/paper/2402.08682/) (2024)
- [VideoMV: Consistent Multi-View Generation Based on Large Video Generative Model](https://mlsanity.com/paper/2403.12010/) (2024)
- [Score Distillation via Reparametrized DDIM](https://mlsanity.com/paper/2405.15891/) (2024)
- [AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation](https://mlsanity.com/paper/2410.07164/) (2024)
- [Collaborative Control for Geometry-Conditioned PBR Image Generation](https://mlsanity.com/paper/2402.05919/) (2024)
- [3D Mesh Editing using Masked LRMs](https://mlsanity.com/paper/2412.08641/) (2024)
- [LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields](https://mlsanity.com/paper/2504.20026/) (2025)
- [Text-to-3D Shape Generation](https://mlsanity.com/paper/2403.13289/) (2024)
- [A Survey on 3D Human Avatar Modeling -- From Reconstruction to Generation](https://mlsanity.com/paper/2406.04253/) (2024)
- [Envision3D: One Image to 3D with Anchor Views Interpolation](https://mlsanity.com/paper/2403.08902/) (2024)
- [RecDreamer: Consistent Text-to-3D Generation via Uniform Score Distillation](https://mlsanity.com/paper/2502.12640/) (2025)
- [A General Framework to Boost 3D GS Initialization for Text-to-3D Generation by Lexical Richness](https://mlsanity.com/paper/2408.01269/) (2024)
- [Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data](https://mlsanity.com/paper/2503.21694/) (2025)
- [PlacidDreamer: Advancing Harmony in Text-to-3D Generation](https://mlsanity.com/paper/2407.13976/) (2024)
- [Apply Hierarchical-Chain-of-Generation to Complex Attributes Text-to-3D Generation](https://mlsanity.com/paper/2505.05505/) (2025)
- [Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion](https://mlsanity.com/paper/2402.16305/) (2024)
- [RewardSDS: Aligning Score Distillation via Reward-Weighted Sampling](https://mlsanity.com/paper/2503.09601/) (2025)
- [Walking the Schr"odinger Bridge: A Direct Trajectory for Text-to-3D Generation](https://mlsanity.com/paper/2511.05609/) (2025)
- [Text-to-3D Generation using Jensen-Shannon Score Distillation](https://mlsanity.com/paper/2503.10660/) (2025)
- [Mean-Shift Distillation for Diffusion Mode Seeking](https://mlsanity.com/paper/2502.15989/) (2025)
- [A Gradient Flow Approach to Solving Inverse Problems with Latent Diffusion Models](https://mlsanity.com/paper/2509.19276/) (2025)
- [Variance Reduction on the Camera Axis: Multi-View Score Distillation for 3D](https://mlsanity.com/paper/2606.29964/) (2026)

## Abstract (arXiv)

Score distillation has emerged as one of the most prevalent approaches for text-to-3D asset synthesis. Essentially, score distillation updates 3D parameters by lifting and back-propagating scores averaged over different views. In this paper, we reveal that the gradient estimation in score distillation is inherent to high variance. Through the lens of variance reduction, the effectiveness of SDS and VSD can be interpreted as applications of various control variates to the Monte Carlo estimator of the distilled score. Motivated by this rethinking and based on Stein's identity, we propose a more general solution to reduce variance for score distillation, termed Stein Score Distillation (SSD). SSD incorporates control variates constructed by Stein identity, allowing for arbitrary baseline functions. This enables us to include flexible guidance priors and network architectures to explicitly optimize for variance reduction. In our experiments, the overall pipeline, dubbed SteinDreamer, is implemented by instantiating the control variate with a monocular depth estimator. The results suggest that SSD can effectively reduce the distillation variance and consistently improve visual quality for both object- and scene-level generation. Moreover, we demonstrate that SteinDreamer achieves faster convergence than existing methods due to more stable gradient updates.