mlsanityData to 7 Oct 2026

Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures

arXiv 2211.07600 ↗First posted 14 Nov 2022cs.CV cs.GRcited by 506

Not on the map. Why?

Cited by 215 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Text-guided image generation has progressed rapidly in recent years, inspiring major breakthroughs in text-guided shape generation. Recently, it has been shown that using score distillation, one can successfully text-guide a NeRF model to generate a 3D object. We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models, which apply the entire diffusion process in a compact latent space of a pretrained autoencoder. As NeRFs operate in image space, a naive solution for guiding them with latent score distillation would require encoding to the latent space at each guidance step. Instead, we propose to bring the NeRF to the latent space, resulting in a Latent-NeRF. Analyzing our Latent-NeRF, we show that while Text-to-3D models can generate impressive results, they are inherently unconstrained and may lack the ability to guide or enforce a specific 3D structure. To assist and direct the 3D generation, we propose to guide our Latent-NeRF using a Sketch-Shape: an abstract geometry that defines the coarse structure of the desired object. Then, we present means to integrate such a constraint directly into a Latent-NeRF. This unique combination of text and shape guidance allows for increased control over the generation process. We also show that latent score distillation can be successfully applied directly on 3D meshes. This allows for generating high-quality textures on a given geometry. Our experiments validate the power of our different forms of guidance and the efficiency of using latent rendering. Implementation is available at https://github.com/eladrich/latent-nerf

Key facts. Cited by 506 papers in the map (data through 7 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 14 Nov 2022.

Builds on

43 of 58 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
31 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  5. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  6. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  8. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  9. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  10. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  11. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  12. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  13. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  14. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  15. 2022Blended Latent Diffusioncited by 430
  16. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  17. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  18. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  19. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  20. 2022Novel View Synthesis with Diffusion Modelscited by 299
  21. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  22. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  23. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  24. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  25. 2019SDM-NET: Deep Generative Network for Structured Deformable Meshcited by 113
  26. 2019Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Researchcited by 98
  27. 2022TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decompositioncited by 83
  28. 2018Point Cloud GANcited by 79
  29. 2021CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fieldscited by 55
  30. 2020DSG-Net: Learning Disentangled Structure and Geometry for 3D Shape Generationcited by 31
  31. 2022SPAGHETTI: Editing Implicit Shapes Through Part Aware Generationcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 166; 2024: 201; 2025: 100; 2026: 36.

506 papers.

Sort:

By area:

  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  2. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  3. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  4. 2023Adversarial Diffusion Distillationcited by 741
  5. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  6. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  7. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  8. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  9. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  10. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  11. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  12. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
38 more
  1. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  2. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  3. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  4. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  5. 2023Text-to-3D using Gaussian Splattingcited by 299
  6. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  7. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  8. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  9. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  10. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  11. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  12. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  13. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 232
  14. 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Modelscited by 179
  15. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  16. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  17. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generationcited by 162
  18. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  19. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Modelscited by 154
  20. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 154
  21. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151
  22. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  23. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  24. 2024MeshLRM: Large Reconstruction Model for High-Quality Meshescited by 138
  25. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  26. 2023Text-to-3D with Classifier Score Distillationcited by 127
  27. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  28. 2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
  29. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 122
  30. 2023Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Modelscited by 122
  31. 2023Delta Denoising Scorecited by 120
  32. 2024CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refinercited by 120
  33. 2023Vox-E: Text-guided Voxel Editing of 3D Objectscited by 114
  34. 2022VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Modelscited by 112
  35. 2023TADA! Text to Animatable Digital Avatarscited by 109
  36. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  37. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  38. 2023DreamHuman: Animatable 3D Avatars from Textcited by 102

50 most cited of 506 All 506 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20233D-CLFusion: Fast Text-to-3D Rendering with Contrastive Latent Diffusion
  2. 2025Advancing Text-to-3D Generation with Linearized Lookahead Variational Score Distillation
  3. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models
  4. 2024SketchDream: Sketch-based Text-to-3D Generation and Editing
  5. 2026Structured 3D Latents Are Surprisingly Powerful: Unleashing Generalizable Style with 2D Diffusion
  6. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts
  7. 2023PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion
  8. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion