mlsanityData to 14 Oct 2026

ET3D: Efficient Text-to-3D Generation via Multi-View Distillation

arXiv 2311.15561 ↗First posted 27 Nov 2023cs.CVcited by 6

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Recent breakthroughs in text-to-image generation has shown encouraging results via large generative models. Due to the scarcity of 3D assets, it is hardly to transfer the success of text-to-image generation to that of text-to-3D generation. Existing text-to-3D generation methods usually adopt the paradigm of DreamFusion, which conducts per-asset optimization by distilling a pretrained text-to-image diffusion model. The generation speed usually ranges from several minutes to tens of minutes per 3D asset, which degrades the user experience and also imposes a burden to the service providers due to the high computational budget. In this work, we present an efficient text-to-3D generation method, which requires only around 8 ms to generate a 3D asset given the text prompt on a consumer graphic card. The main insight is that we exploit the images generated by a large pre-trained text-to-image diffusion model, to supervise the training of a text conditioned 3D generative adversarial network. Once the network is trained, we are able to efficiently generate a 3D asset via a single forward pass. Our method requires no 3D training data and provides an alternative approach for efficient text-to-3D generation by distilling pre-trained image diffusion models.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 66 of its 71 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

66 of 71 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D4 Oct
  2. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  3. 2023DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation21 Jun
  4. 2023StyleAvatar3D: Leveraging Image-Text Diffusion Models for High-Fidelity 3D Avatar Generation30 May
  5. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance30 May
  6. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  7. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  8. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  9. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  10. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  11. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  12. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions24 Nov
4 more
  1. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  2. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  3. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  4. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  6. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  7. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  8. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  10. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  11. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  12. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
38 more
  1. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  2. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  3. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
  4. 2018Which Training Methods for GANs do actually Converge?cited by 679
  5. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  6. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  7. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  8. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 488
  9. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  10. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  11. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  12. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  13. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  14. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  15. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 212
  16. 20193D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutionscited by 173
  17. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  18. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  19. 2021CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesiscited by 149
  20. 2022VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Gridscited by 133
  21. 2022Neural Wavelet-domain Diffusion for 3D Shape Generationcited by 132
  22. 2016Generative and Discriminative Voxel Modeling with Convolutional Neural Networkscited by 128
  23. 2018Visual Object Networks: Image Generation with Disentangled 3D Representationcited by 124
  24. 20163D Shape Induction from 2D Views of Multiple Objectscited by 121
  25. 2019SDM-NET: Deep Generative Network for Structured Deformable Meshcited by 113
  26. 2019Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesiscited by 104
  27. 2021GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worldscited by 98
  28. 2020Discrete Point Flow Networks for Efficient Point Cloud Generationcited by 87
  29. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  30. 2017Variational Autoencoders for Deforming 3D Mesh Modelscited by 70
  31. 2019Unsupervised Generative 3D Shape Learning from Natural Imagescited by 57
  32. 2022FLAG: Flow-based 3D Avatar Generation from Sparse Observationscited by 55
  33. 2019C-Flow: Conditional Generative Flow Models for Images and 3D Point Cloudscited by 50
  34. 2021Learning Generative Models of Textured 3D Meshes from Real-World Imagescited by 44
  35. 2022CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshescited by 41
  36. 2018Multi-chart Generative Surface Modelingcited by 26
  37. 2019Synthesizing 3D Shapes from Silhouette Image Collections using Multi-projection Generative Adversarial Networkscited by 14
  38. 2018Structure-Aware Shape Synthesiscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4.

6 papers.

  1. 2024LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesiscited by 39
  2. 2024Hash3D: Training-free Acceleration for 3D Generationcited by 19
  3. 2024DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstructioncited by 16
  4. 2024A Survey On Text-to-3D Contents Generation In The Wildcited by 14
  5. 2023DiverseDream: Diverse Text-to-3D Synthesis with Augmented Text Embeddingcited by 1
  6. 2023Text-Image Conditioned Diffusion for Consistent Text-to-3D Generationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesis
  2. 2024BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion
  3. 2024DreamFlow: High-Quality Text-to-3D Generation by Approximating Probability Flow
  4. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model
  5. 2024ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models
  6. 2024GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space
  7. 2024Grounded Compositional and Diverse Text-to-3D with Pretrained Multi-View Diffusion Model
  8. 2024Dual3D: Efficient and Consistent Text-to-3D Generation with Dual-mode Multi-view Latent Diffusion