mlsanityData to 14 Oct 2026

DreamFusion: Text-to-3D using 2D Diffusion

arXiv 2209.14988 ↗First posted 29 Sep 2022cs.CV cs.LG stat.MLcited by 3,025

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.

Key facts. Cited by 3,025 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 54 of its 76 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

54 of 76 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  3. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
38 more
  1. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  2. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  4. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  6. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  7. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
  8. 2021Variational Diffusion Modelscited by 1,153
  9. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  10. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
  11. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  12. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 575
  13. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  14. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  15. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  16. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 496
  17. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 488
  18. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  19. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 449
  20. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  21. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  22. 2021Advances in Neural Renderingcited by 365
  23. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 334
  24. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  25. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  26. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  27. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  28. 2020Learning Gradient Fields for Shape Generationcited by 245
  29. 2022Diffusion models as plug-and-play priorscited by 231
  30. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 215
  31. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  32. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  33. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  34. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 98
  35. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  36. 2022Towards Implicit Text-Guided 3D Shape Generationcited by 68
  37. 2021ClipMatrix: Text-controlled Creation of 3D Textured Meshescited by 40
  38. 2020AR-DAE: Towards Unbiased Neural Entropy Gradient Estimationcited by 19

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 57; 2023: 651; 2024: 1022; 2025: 813; 2026: 482.

3,025 papers.

Sort:

By area:

  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  2. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  4. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  5. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  6. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  7. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  8. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 938
  9. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  10. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  11. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  12. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
38 more
  1. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
  2. 2023Adversarial Diffusion Distillationcited by 759
  3. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  4. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  5. 2023Scaffold-GS: Structured 3D Gaussians for View-Adaptive Renderingcited by 695
  6. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
  7. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 687
  8. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 684
  9. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 671
  10. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 645
  11. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 600
  12. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  13. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 587
  14. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 560
  15. 2025Continuous 3D Perception Model with Persistent Statecited by 548
  16. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 521
  17. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  18. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  19. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  20. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  21. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 458
  22. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  23. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  24. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  25. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  26. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 436
  27. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 433
  28. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  29. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  30. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 397
  31. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  32. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 378
  33. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  34. 2023DeepCache: Accelerating Diffusion Models for Freecited by 359
  35. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 356
  36. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 343
  37. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  38. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 335

50 most cited of 3,025 All 3,025 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation
  2. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion
  3. 2022Implementing and Experimenting with Diffusion Models for Text-to-Image Generation
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
  5. 2024Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy