mlsanityData to 7 Oct 2026

Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models

arXiv 2212.14704 ↗First posted 28 Dec 2022cs.CVcited by 151

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Recent CLIP-guided 3D optimization methods, such as DreamFields and PureCLIPNeRF, have achieved impressive results in zero-shot text-to-3D synthesis. However, due to scratch training and random initialization without prior knowledge, these methods often fail to generate accurate and faithful 3D structures that conform to the input text. In this paper, we make the first attempt to introduce explicit 3D shape priors into the CLIP-guided 3D optimization process. Specifically, we first generate a high-quality 3D shape from the input text in the text-to-shape stage as a 3D shape prior. We then use it as the initialization of a neural radiance field and optimize it with the full prompt. To address the challenging text-to-shape generation task, we present a simple yet effective approach that directly bridges the text and image modalities with a powerful text-to-image diffusion model. To narrow the style domain gap between the images synthesized by the text-to-image diffusion model and shape renderings used to train the image-to-shape generator, we further propose to jointly optimize a learnable text prompt and fine-tune the text-to-image diffusion model for rendering-style image generation. Our method, Dream3D, is capable of generating imaginative 3D content with superior visual quality and shape accuracy compared to state-of-the-art methods.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 83 of its 92 references that are in the map. First posted on arXiv 28 Dec 2022.

Builds on

83 of 92 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  2. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  3. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  4. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 79

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
38 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  5. 2015Variational Inference with Normalizing Flowscited by 2,161
  6. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  7. 2016Density estimation using Real NVPcited by 2,029
  8. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  10. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  11. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  12. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  13. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  14. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  15. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  16. 2016Generative Adversarial Text to Image Synthesiscited by 735
  17. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  18. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  19. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  20. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  21. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  22. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  23. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  24. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  25. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  26. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  27. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  28. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  29. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  30. 2017Neural 3D Mesh Renderercited by 437
  31. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  32. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  33. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  34. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 267
  35. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  36. 2020Learning Gradient Fields for Shape Generationcited by 245
  37. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  38. 2021GRAM: Generative Radiance Manifolds for 3D-Aware Image Generationcited by 216

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 69; 2024: 58; 2025: 15; 2026: 9.

151 papers.

Sort:

By area:

  1. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  2. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  3. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  4. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  5. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  6. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  7. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 254
  8. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  9. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  10. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generationcited by 181
  11. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  12. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Modelscited by 154
38 more
  1. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  2. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  3. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidancecited by 109
  4. 2023TADA! Text to Animatable Digital Avatarscited by 109
  5. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusionscited by 108
  6. 2023DemoFusion: Democratising High-Resolution Image Generation With No cited by 105
  7. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  8. 2023GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphscited by 96
  9. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  10. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  11. 2023HumanMAC: Masked Motion Completion for Human Motion Predictioncited by 81
  12. 2023SKED: Sketch-guided Text-based 3D Editingcited by 72
  13. 2023GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learningcited by 63
  14. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  15. 2024SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfercited by 61
  16. 2023CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layoutcited by 57
  17. 2023ViVid-1-to-3: Novel View Synthesis with Video Diffusion Modelscited by 57
  18. 2023T3Bench: Benchmarking Current Progress in Text-to-3D Generationcited by 56
  19. 2023How to Trust Your Diffusion Model: A Convex Optimization Approach to Conformal Risk Controlcited by 50
  20. 2024DiffGS: Functional Gaussian Splatting Diffusioncited by 50
  21. 2023HeadSculpt: Crafting 3D Head Avatars with Textcited by 48
  22. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 47
  23. 2023Point Cloud Pre-training with Diffusion Modelscited by 47
  24. 2023UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generationcited by 47
  25. 2024REPARO: Compositional 3D Assets Generation with Differentiable 3D Layout Alignmentcited by 45
  26. 2024GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modelingcited by 44
  27. 2023CG3D: Compositional Generation for Text-to-3D via Gaussian Splattingcited by 42
  28. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  29. 2024Scaling Diffusion Models to Real-World 3D LiDAR Scene Completioncited by 40
  30. 2023X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidancecited by 37
  31. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  32. 2024SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Viewscited by 35
  33. 2024Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Imagescited by 34
  34. 2023Cascade-Zero123: One Image to Highly Consistent 3D with Self-Prompted Nearby Viewscited by 32
  35. 2024AccDiffusion: An Accurate Method for Higher-Resolution Image Generationcited by 32
  36. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
  37. 2023DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Viewscited by 31
  38. 2025AnimateAnyMesh: A Feed-Forward 4D Foundation Model for Text-Driven Universal Mesh Animationcited by 31

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts
  2. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields
  3. 2022Magic3D: High-Resolution Text-to-3D Content Creation
  4. 2023PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion
  5. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model
  6. 2023DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation
  7. 2023DreamStone: Image as Stepping Stone for Text-Guided 3D Shape Generation
  8. 2023X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation