mlsanityData to 14 Oct 2026

RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D

arXiv 2311.16918 ↗First posted 28 Nov 2023cs.CV cs.AIcited by 168

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Lifting 2D diffusion for 3D generation is a challenging problem due to the lack of geometric prior and the complex entanglement of materials and lighting in natural images. Existing methods have shown promise by first creating the geometry through score-distillation sampling (SDS) applied to rendered surface normals, followed by appearance modeling. However, relying on a 2D RGB diffusion model to optimize surface normals is suboptimal due to the distribution discrepancy between natural images and normals maps, leading to instability in optimization. In this paper, recognizing that the normal and depth information effectively describe scene geometry and be automatically estimated from images, we propose to learn a generalizable Normal-Depth diffusion model for 3D generation. We achieve this by training on the large-scale LAION dataset together with the generalizable image-to-depth and normal prior models. In an attempt to alleviate the mixed illumination effects in the generated materials, we introduce an albedo diffusion model to impose data-driven constraints on the albedo component. Our experiments show that when integrated into existing text-to-3D pipelines, our models significantly enhance the detail richness, achieving state-of-the-art results. Our project page is https://aigc3d.github.io/richdreamer/.

Key facts. Cited by 168 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 88 of its 99 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

88 of 99 references.

Closest ancestors · 43

Same direction, newest first

  1. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model23 Oct
  2. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  3. 2023Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts18 Oct
  4. 2023Consistent123: Improve Consistency for One Image to 3D Object Synthesis12 Oct
  5. 2023Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models4 Oct
  6. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D4 Oct
  7. 2023HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation2 Oct
  8. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  9. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  10. 2023HoloFusion: Towards Photo-realistic 3D Generative Modeling28 Aug
  11. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesis22 Aug
  12. 2023MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR18 Aug
31 more
  1. 2023HD-Fusion: Detailed Text-to-3D Generation Leveraging Multiple Noise Estimation30 Jul
  2. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
  3. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  4. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data13 Jun
  5. 2023HeadSculpt: Crafting 3D Head Avatars with Text5 Jun
  6. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  7. 2023LDM3D: Latent Diffusion Model for 3D18 May
  8. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts24 Apr
  9. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction13 Apr
  10. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Models5 Apr
  11. 2023Consistent View Synthesis with Pose-Guided Diffusion Models30 Mar
  12. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior24 Mar
  13. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  14. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  15. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation14 Mar
  16. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
  17. 2023TEXTure: Text-Guided Texturing of 3D Shapes3 Feb
  18. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models26 Jan
  19. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models28 Dec
  20. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  21. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  22. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors6 Dec
  23. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  24. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  25. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Views29 Nov
  26. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  27. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation17 Nov
  28. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  29. 2022Novel View Synthesis with Diffusion Models6 Oct
  30. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  31. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2021Vision Transformers for Dense Predictioncited by 1,803
33 more
  1. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  2. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  3. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  4. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  5. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  6. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  7. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  8. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 460
  9. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  10. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 441
  11. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  12. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 366
  13. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 326
  14. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  15. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 288
  16. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  17. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  18. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
  19. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 220
  20. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  21. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  22. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 199
  23. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 180
  24. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  25. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  26. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  27. 2021Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimationcited by 116
  28. 2023TADA! Text to Animatable Digital Avatarscited by 110
  29. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  30. 2023DreamWaltz: Make a Scene with Complex 3D Animatable Avatarscited by 82
  31. 2023DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidancecited by 80
  32. 2023Pushing the Limits of 3D Shape Generation at Scalecited by 11
  33. 2017Depth Super-Resolution Meets Uncalibrated Photometric Stereocited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 80; 2025: 56; 2026: 29.

168 papers.

Sort:

By area:

  1. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  2. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 286
  3. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 215
  4. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
  5. 2024StableNormal: Reducing Diffusion Variance for Stable and Sharp Normalcited by 154
  6. 2024L4GM: Large 4D Gaussian Reconstruction Modelcited by 144
  7. 2025Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal Bridgingcited by 99
  8. 2024Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoderscited by 98
  9. 2024SAMPart3D: Segment Any Part in 3D Objectscited by 97
  10. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 92
  11. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 87
  12. 2024IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generationcited by 84
38 more
  1. 2024Fine-Tuning Image-Conditional Diffusion Models is Easier than You Thinkcited by 83
  2. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priorscited by 74
  3. 2025Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstructioncited by 67
  4. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
  5. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  6. 2025DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learningcited by 65
  7. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 62
  8. 2024Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstructioncited by 56
  9. 2024Animate3D: Animating Any 3D Model with Multi-view Video Diffusioncited by 56
  10. 2024LaRa: Efficient Large-Baseline Radiance Fieldscited by 55
  11. 2024MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Modelcited by 55
  12. 2024Fast Feedforward 3D Gaussian Splatting Compressioncited by 52
  13. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 50
  14. 2023UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generationcited by 47
  15. 2024VideoMV: Consistent Multi-View Generation Based on Large Video Generative Modelcited by 45
  16. 2024DressCode: Autoregressively Sewing and Generating Garments from Text Guidancecited by 44
  17. 2025Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modelingcited by 43
  18. 2024Vivid-ZOO: Multi-View Video Generation with Diffusion Modelcited by 40
  19. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAEcited by 39
  20. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 38
  21. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 38
  22. 20243DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generationcited by 37
  23. 2024GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generationcited by 36
  24. 2024Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cyclecited by 36
  25. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  26. 2024Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Imagescited by 35
  27. 2025ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generationcited by 33
  28. 2025AnimateAnyMesh: A Feed-Forward 4D Foundation Model for Text-Driven Universal Mesh Animationcited by 32
  29. 2025Decompositional Neural Scene Reconstruction with Generative Diffusion Priorcited by 31
  30. 2024Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamicscited by 28
  31. 2024ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillationcited by 27
  32. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  33. 2025DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundnesscited by 25
  34. 2024Real3D: Scaling Up Large Reconstruction Models with Real-World Imagescited by 24
  35. 2024MVGamba: Unify 3D Content Generation as State Space Sequence Modelingcited by 23
  36. 2023HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillationcited by 22
  37. 2024IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminationscited by 22
  38. 2024TetSphere Splatting: Representing High-Quality Geometry with Lagrangian Volumetric Meshescited by 21

50 most cited of 168 All 168 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023HyperDreamer: Hyper-Realistic 3D Content Generation and Editing from a Single Image
  2. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion
  3. 2023Breathing New Life into 3D Assets with Generative Repainting
  4. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models
  5. 2024BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion
  6. 2024Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions
  7. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields
  8. 2023Text-guided High-definition Consistency Texture Model