mlsanityData to 14 Oct 2026

Zero-Shot Text-Guided Object Generation with Dream Fields

arXiv 2112.01455 ↗First posted 2 Dec 2021cs.CV cs.AI cs.GRcited by 513

Not on the map. Why?

Cited by 176 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We combine neural rendering with multi-modal image and text representations to synthesize diverse 3D objects solely from natural language descriptions. Our method, Dream Fields, can generate the geometry and color of a wide range of objects without 3D supervision. Due to the scarcity of diverse, captioned 3D data, prior methods only generate objects from a handful of categories, such as ShapeNet. Instead, we guide generation with image-text models pre-trained on large datasets of captioned images from the web. Our method optimizes a Neural Radiance Field from many camera views so that rendered images score highly with a target caption according to a pre-trained CLIP model. To improve fidelity and visual quality, we introduce simple geometric priors, including sparsity-inducing transmittance regularization, scene bounds, and new MLP architectures. In experiments, Dream Fields produce realistic, multi-view consistent object geometry and color from a variety of natural language captions.

Key facts. Cited by 513 papers in the map (data through 14 Oct 2026). Builds on 46 of its 67 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

46 of 67 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
34 more
  1. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  2. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  3. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  4. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  5. 2021Volume Rendering of Neural Implicit Surfacescited by 996
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  7. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 861
  8. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  9. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  10. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
  11. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  12. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  13. 2019On Variational Bounds of Mutual Informationcited by 559
  14. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  15. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  16. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  17. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  18. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  19. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  20. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  21. 2020Learning Gradient Fields for Shape Generationcited by 245
  22. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 242
  23. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 187
  24. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  25. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  26. 2021CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesiscited by 149
  27. 2021Unconstrained Scene Generation with Locally Conditioned Radiance Fieldscited by 130
  28. 2021Learning to Compose Visual Relationscited by 72
  29. 2020Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flowscited by 61
  30. 2018Learning to Reconstruct Shapes from Unseen Classescited by 60
  31. 2021ClipMatrix: Text-controlled Creation of 3D Textured Meshescited by 40
  32. 2015Text to 3D Scene Generation with Rich Lexical Groundingcited by 36
  33. 2021StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesiscited by 15
  34. 2021Evolving Evocative 2D Views of Generated 3D Objectscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 54; 2023: 227; 2024: 163; 2025: 46; 2026: 22.

513 papers.

Sort:

By area:

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  5. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  6. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  7. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  8. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  9. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
  10. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 687
  11. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  12. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 587
38 more
  1. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 560
  2. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  3. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  4. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  5. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 433
  6. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 397
  7. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  8. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 378
  9. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  10. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 343
  11. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
  12. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  13. 2023FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularizationcited by 311
  14. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  15. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 301
  16. 2023Text-to-3D using Gaussian Splattingcited by 299
  17. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  18. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  19. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  20. 2023FSGS: Real-Time Few-shot View Synthesis using Gaussian Splattingcited by 279
  21. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  22. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 259
  23. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  24. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  25. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  26. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
  27. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 214
  28. 2023Text-To-4D Dynamic Scene Generationcited by 205
  29. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  30. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  31. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  32. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  33. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generationcited by 183
  34. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 180
  35. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  36. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  37. 2022SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Imagecited by 165
  38. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generationcited by 162

50 most cited of 513 All 513 (JSON)