mlsanityData to 7 Oct 2026

Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation

arXiv 2303.13873 ↗First posted 24 Mar 2023cs.CV cs.AIcited by 684

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Automatic 3D content creation has achieved rapid progress recently due to the availability of pre-trained, large language models and image diffusion models, forming the emerging topic of text-to-3D content creation. Existing text-to-3D methods commonly use implicit scene representations, which couple the geometry and appearance via volume rendering and are suboptimal in terms of recovering finer geometries and achieving photorealistic rendering; consequently, they are less effective for generating high-quality 3D assets. In this work, we propose a new method of Fantasia3D for high-quality text-to-3D content creation. Key to Fantasia3D is the disentangled modeling and learning of geometry and appearance. For geometry learning, we rely on a hybrid scene representation, and propose to encode surface normal extracted from the representation as the input of the image diffusion model. For appearance modeling, we introduce the spatially varying bidirectional reflectance distribution function (BRDF) into the text-to-3D task, and learn the surface material for photorealistic rendering of the generated surface. Our disentangled framework is more compatible with popular graphics engines, supporting relighting, editing, and physical simulation of the generated 3D assets. We conduct thorough experiments that show the advantages of our method over existing ones under different text-to-3D task settings. Project page and source codes: https://fantasia3d.github.io/.

Key facts. Cited by 684 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 33 of its 44 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

33 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  2. 20223D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models1 Dec
  3. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  4. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  5. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 28

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  8. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  9. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  10. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  11. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  12. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
16 more
  1. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  2. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  3. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  4. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  5. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 439
  6. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  7. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 363
  8. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  9. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  10. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  11. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  12. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  13. 2018Single-Image SVBRDF Capture with a Rendering-Aware Deep Networkcited by 116
  14. 2020Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Renderingcited by 101
  15. 2022TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decompositioncited by 83
  16. 2021DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderercited by 52

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 158; 2024: 297; 2025: 158; 2026: 71.

684 papers.

Sort:

By area:

  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  2. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  3. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  4. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  5. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  6. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  7. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  8. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  9. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  10. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  11. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  12. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
38 more
  1. 2023Text-to-3D using Gaussian Splattingcited by 299
  2. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  3. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  4. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  5. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  6. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  7. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  8. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generationcited by 256
  9. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  10. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
  11. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 191
  12. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  13. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  14. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  15. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  16. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Modelscited by 154
  17. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 154
  18. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  19. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  20. 2024MeshLRM: Large Reconstruction Model for High-Quality Meshescited by 138
  21. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3Dcited by 134
  22. 2024Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Imagecited by 132
  23. 2023Text-to-3D with Classifier Score Distillationcited by 127
  24. 2023Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Videocited by 124
  25. 2023Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Modelscited by 122
  26. 2024CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refinercited by 120
  27. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 120
  28. 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 118
  29. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 116
  30. 2023Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusioncited by 114
  31. 2023Animate124: Animating One Image to 4D Dynamic Scenecited by 113
  32. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidancecited by 109
  33. 2023TADA! Text to Animatable Digital Avatarscited by 109
  34. 2023ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Imagecited by 109
  35. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  36. 2023TeCH: Text-guided Reconstruction of Lifelike Clothed Humanscited by 104
  37. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  38. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generationcited by 104

50 most cited of 684 All 684 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior
  2. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields
  3. 2023Breathing New Life into 3D Assets with Generative Repainting
  4. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts
  5. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models
  6. 2025Syn3DTxt: Embedding 3D Cues for Scene Text Generation
  7. 2026SIC3D: Style Image Conditioned Text-to-3D Gaussian Splatting Generation
  8. 2023Control3D: Towards Controllable Text-to-3D Generation