mlsanityData to 7 Oct 2026

UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation

arXiv 2312.08754 ↗First posted 14 Dec 2023cs.CVcited by 47

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Recent advancements in text-to-3D generation technology have significantly advanced the conversion of textual descriptions into imaginative well-geometrical and finely textured 3D objects. Despite these developments, a prevalent limitation arises from the use of RGB data in diffusion or reconstruction models, which often results in models with inherent lighting and shadows effects that detract from their realism, thereby limiting their usability in applications that demand accurate relighting capabilities. To bridge this gap, we present UniDream, a text-to-3D generation framework by incorporating unified diffusion priors. Our approach consists of three main components: (1) a dual-phase training process to get albedo-normal aligned multi-view diffusion and reconstruction models, (2) a progressive generation procedure for geometry and albedo-textures based on Score Distillation Sample (SDS) using the trained reconstruction and diffusion models, and (3) an innovative application of SDS for finalizing PBR generation while keeping a fixed albedo based on Stable Diffusion model. Extensive evaluations demonstrate that UniDream surpasses existing methods in generating 3D objects with clearer albedo textures, smoother surfaces, enhanced realism, and superior relighting capabilities.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 38 of its 61 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

38 of 61 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior11 Dec
  2. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D28 Nov
  3. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model10 Nov
  4. 2023Text-to-3D with Classifier Score Distillation30 Oct
  5. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  6. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  7. 2023MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR18 Aug
  8. 2023Points-to-3D: Bridging the Gap between Sparse Points and Shape-Controllable Text-to-3D Generation26 Jul
  9. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  10. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  11. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  12. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
4 more
  1. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models28 Dec
  2. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  3. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  4. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  7. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  8. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  9. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  10. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  11. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  12. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 493
10 more
  1. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  2. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  3. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 363
  4. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  5. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  6. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  7. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  8. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  9. 2022Neural Wavelet-domain Diffusion for 3D Shape Generationcited by 132
  10. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 129

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 15; 2026: 2.

47 papers.

Sort:

By area:

  1. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  2. 2025TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Modelscited by 204
  3. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  4. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 120
  5. 2024Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attentioncited by 99
  6. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 88
  7. 2025SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modelingcited by 76
  8. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
  9. 2024SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglementcited by 65
  10. 2024GVGEN: Text-to-3D Generation with Volumetric Representationcited by 51
  11. 2024DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Modelscited by 50
  12. 2024Director3D: Real-world Camera Trajectory and 3D Scene Generation from Textcited by 50
35 more
  1. 2024A Comprehensive Survey on 3D Content Generationcited by 28
  2. 2025MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusioncited by 28
  3. 2024ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillationcited by 27
  4. 2024AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animationcited by 25
  5. 2024Collaborative Control for Geometry-Conditioned PBR Image Generationcited by 22
  6. 2024TAR3D: Creating High-Quality 3D Assets via Next-Part Predictioncited by 16
  7. 2024DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstructioncited by 15
  8. 2024RoomTex: Texturing Compositional Indoor Scenes via Iterative Inpaintingcited by 14
  9. 2024A Survey on 3D Human Avatar Modeling -- From Reconstruction to Generationcited by 13
  10. 2024Envision3D: One Image to 3D with Anchor Views Interpolationcited by 12
  11. 2024Large Point-to-Gaussian Model for Image-to-3D Generationcited by 10
  12. 2024Boosting 3D Object Generation through PBR Materialscited by 10
  13. 2024SuperMat: Physically Consistent PBR Material Estimation at Interactive Ratescited by 9
  14. 2025MuMA: 3D PBR Texturing via Multi-Channel Multi-View Generation and Agentic Post-Processingcited by 8
  15. 2024DreamCraft3D++: Efficient Hierarchical 3D Generation with Multi-Plane Reconstruction Modelcited by 7
  16. 2024ARM: Appearance Reconstruction Model for Relightable 3D Generationcited by 7
  17. 2025Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Datacited by 7
  18. 2025SViM3D: Stable Video Material Diffusion for Single Image 3D Generationcited by 7
  19. 2025NaTex: Seamless Texture Generation as Latent Color Diffusioncited by 7
  20. 2024MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assetscited by 6
  21. 2025Apply Hierarchical-Chain-of-Generation to Complex Attributes Text-to-3D Generationcited by 5
  22. 2025SegmentDreamer: Towards High-fidelity Text-to-3D Synthesis with Segmented Consistency Trajectory Distillationcited by 5
  23. 2024Jointly Generating Multi-view Consistent PBR Textures using Collaborative Controlcited by 4
  24. 2025NeAR: Coupled Neural Asset-Renderer Stackcited by 4
  25. 2024Meta 3D Gencited by 3
  26. 2024Magic-Boost: Boost 3D Generation with Multi-View Conditioned Diffusioncited by 2
  27. 2024COMOGen: A Controllable Text-to-3D Multi-object Generation Frameworkcited by 2
  28. 2024MVLight: Relightable Text-to-3D Generation via Light-conditioned Multi-View Diffusioncited by 2
  29. 2025Large Material Gaussian Model for Relightable 3D Generationcited by 2
  30. 2025MatMart: Material Reconstruction of 3D Objects via Diffusioncited by 2
  31. 2025LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Contextcited by 2
  32. 2026Hitem3D 2.0: Multi-View Guided Native 3D Texture Generationcited by 2
  33. 2024HOTS3D: Hyper-Spherical Optimal Transport for Semantic Alignment of Text-to-3D Generationcited by 1
  34. 2026Interact3D: Compositional 3D Generation of Interactive Objectscited by 1
  35. 2025DreamLifting: A Plug-in Module Lifting MV Diffusion Models for 3D Asset Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields
  2. 2023Breathing New Life into 3D Assets with Generative Repainting
  3. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation
  4. 2024GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generation
  5. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors
  6. 2023Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation
  7. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts
  8. 2023X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation