mlsanityData to 7 Oct 2026

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

arXiv 2305.16213 ↗First posted 25 May 2023cs.LG cs.CVcited by 1,207

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512×512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page and codes: https://ml.cs.tsinghua.edu.cn/prolificdreamer/

Key facts. Cited by 1,207 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 44 of its 57 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

44 of 57 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields19 May
  2. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts24 Apr
  3. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  5. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation14 Mar
  6. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  8. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
24 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  3. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  4. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  5. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  6. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  8. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  9. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  10. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  11. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  12. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  13. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  14. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  15. 2022Diffusion models as plug-and-play priorscited by 230
  16. 2019Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Researchcited by 98
  17. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  18. 2018A Stein variational Newton methodcited by 49
  19. 2018A Unified Particle-Optimization Framework for Scalable Bayesian Samplingcited by 44
  20. 2019Function Space Particle Optimization for Bayesian Neural Networkscited by 41
  21. 2017Learning to Draw Samples with Amortized Stein Variational Gradient Descentcited by 26
  22. 2022Particle-based Variational Inference with Preconditioned Functional Gradient Flowcited by 20
  23. 2021Neural Variational Gradient Descentcited by 19
  24. 2018Understanding and Accelerating Particle-Based Variational Inferencecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 172; 2024: 458; 2025: 357; 2026: 220.

1,207 papers.

Sort:

By area:

  1. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  2. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  3. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  4. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  5. 2023Adversarial Diffusion Distillationcited by 741
  6. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  7. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  8. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  9. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  10. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  11. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 516
  12. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
38 more
  1. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  2. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  3. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  4. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  5. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  6. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  7. 2023SeeSR: Towards Semantics-Aware Real-World Image Super-Resolutioncited by 311
  8. 2023Text-to-3D using Gaussian Splattingcited by 299
  9. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  10. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  11. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  12. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  13. 2024Diffusion Models Are Real-Time Game Enginescited by 282
  14. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  15. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  16. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  17. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 258
  18. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  19. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generationcited by 256
  20. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  21. 2024One-Step Effective Diffusion Network for Real-World Image Super-Resolutioncited by 243
  22. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 228
  23. 2024TripoSR: Fast 3D Object Reconstruction from a Single Imagecited by 227
  24. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
  25. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
  26. 2024Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generationcited by 211
  27. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  28. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 183
  29. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  30. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  31. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  32. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  33. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  34. 2024Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesiscited by 153
  35. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  36. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  37. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  38. 2024L4GM: Large 4D Gaussian Reconstruction Modelcited by 143

50 most cited of 1,207 All 1,207 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024PlacidDreamer: Advancing Harmony in Text-to-3D Generation
  2. 2023Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generation
  3. 2023Collaborative Score Distillation for Consistent Visual Synthesis
  4. 2024Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models