mlsanityData to 7 Oct 2026

Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation

arXiv 2303.07937 ↗First posted 14 Mar 2023cs.CVcited by 133

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Text-to-3D generation has shown rapid progress in recent days with the advent of score distillation, a methodology of using pretrained text-to-2D diffusion models to optimize neural radiance field (NeRF) in the zero-shot setting. However, the lack of 3D awareness in the 2D diffusion models destabilizes score distillation-based methods from reconstructing a plausible 3D scene. To address this issue, we propose 3DFuse, a novel framework that incorporates 3D awareness into pretrained 2D diffusion models, enhancing the robustness and 3D consistency of score distillation-based methods. We realize this by first constructing a coarse 3D structure of a given text prompt and then utilizing projected, view-specific depth map as a condition for the diffusion model. Additionally, we introduce a training strategy that enables the 2D diffusion model learns to handle the errors and sparsity within the coarse 3D structure for robust generation, as well as a method for ensuring semantic consistency throughout all viewpoints of the scene. Our framework surpasses the limitations of prior arts, and has significant implications for 3D consistent generation of 2D diffusion models.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 65 of its 84 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

65 of 84 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models28 Dec
  2. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  3. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors6 Dec
  4. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  5. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Views29 Nov
  6. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  7. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  8. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  7. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2021Vision Transformers for Dense Predictioncited by 1,789
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  12. 2017End-to-end Recovery of Human Shape and Posecited by 883
  13. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  14. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  15. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  16. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  17. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  18. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  19. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  20. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  21. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  22. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  23. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
  24. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  25. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  26. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  27. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 267
  28. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  29. 2022Dual Diffusion Implicit Bridges for Image-to-Image Translationcited by 212
  30. 2017Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputscited by 189
  31. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  32. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidancecited by 151
  33. 20163D Shape Induction from 2D Views of Multiple Objectscited by 121
  34. 2019PQ-NET: A Generative Part Seq2Seq Network for 3D Shapescited by 118
  35. 2019SDM-NET: Deep Generative Network for Structured Deformable Meshcited by 113
  36. 2020Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimationcited by 113
  37. 2022SDF-StyleGAN: Implicit SDF-Based StyleGAN for 3D Shape Generationcited by 92
  38. 2023Multiview Compressive Coding for 3D Reconstructioncited by 71

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 61; 2024: 54; 2025: 15; 2026: 3.

133 papers.

Sort:

By area:

  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  2. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  3. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  4. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  5. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  6. 2023Text-to-3D using Gaussian Splattingcited by 299
  7. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  8. 2023FSGS: Real-Time Few-shot View Synthesis using Gaussian Splattingcited by 278
  9. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  10. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  11. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  12. 2023Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Videocited by 124
38 more
  1. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 122
  2. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117
  3. 2024Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attentioncited by 99
  4. 2024FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulationcited by 99
  5. 2024GES: Generalized Exponential Splatting for Efficient Radiance Field Renderingcited by 92
  6. 2023Anything-3D: Towards Single-view Anything Reconstruction in the Wildcited by 91
  7. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 88
  8. 2023Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Modelscited by 79
  9. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  10. 2023Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priorscited by 64
  11. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesiscited by 63
  12. 2024SPAD : Spatially Aware Multiview Diffuserscited by 62
  13. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  14. 2024SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfercited by 61
  15. 2023Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraintscited by 59
  16. 2023ViVid-1-to-3: Novel View Synthesis with Video Diffusion Modelscited by 57
  17. 2024ViewDiff: 3D-Consistent Image Generation with Text-to-Image Modelscited by 57
  18. 2023T3Bench: Benchmarking Current Progress in Text-to-3D Generationcited by 56
  19. 2024MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Modelcited by 55
  20. 2024AGG: Amortized Generative 3D Gaussians for Single Image to 3Dcited by 51
  21. 2023HeadSculpt: Crafting 3D Head Avatars with Textcited by 48
  22. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 47
  23. 2023PERF: Panoramic Neural Radiance Field from a Single Panoramacited by 45
  24. 2024REPARO: Compositional 3D Assets Generation with Differentiable 3D Layout Alignmentcited by 45
  25. 2024Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Videocited by 44
  26. 2024DressCode: Autoregressively Sewing and Generating Garments from Text Guidancecited by 43
  27. 2023CG3D: Compositional Generation for Text-to-3D via Gaussian Splattingcited by 42
  28. 2023Matte Anything: Interactive Natural Image Matting with Segment Anything Modelscited by 39
  29. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  30. 2023Alchemist: Parametric Control of Material Properties with Diffusion Modelscited by 38
  31. 2023MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusioncited by 37
  32. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  33. 2023Taming Mode Collapse in Score Distillation for Text-to-3D Generationcited by 35
  34. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  35. 2024SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Viewscited by 35
  36. 2023Multi-Architecture Multi-Expert Diffusion Modelscited by 34
  37. 2023Disentangled Clothed Avatar Generation from Text Descriptionscited by 33
  38. 2023Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generationcited by 32

50 most cited of 133 All 133 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models
  2. 2024BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion
  3. 2024JointDreamer: Ensuring Geometry Consistency and Text Congruence in Text-to-3D Generation via Joint Score Distillation
  4. 2025Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generation
  5. 2023UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation
  6. 2025Dive3D: Diverse Distillation-based Text-to-3D Generation via Score Implicit Matching
  7. 2023MVDream: Multi-view Diffusion for 3D Generation
  8. 2023StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D