mlsanityData to 7 Oct 2026

Understanding Pure CLIP Guidance for Voxel Grid NeRF Models

arXiv 2209.15172 ↗First posted 30 Sep 2022cs.CV cs.GR cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

We explore the task of text to 3D object generation using CLIP. Specifically, we use CLIP for guidance without access to any datasets, a setting we refer to as pure CLIP guidance. While prior work has adopted this setting, there is no systematic study of mechanics for preventing adversarial generations within CLIP. We illustrate how different image-based augmentations prevent the adversarial generation problem, and how the generated results are impacted. We test different CLIP model architectures and show that ensembling different models for guidance can prevent adversarial generations within bigger models and generate sharper results. Furthermore, we implement an implicit voxel grid model to show how neural networks provide an additional layer of regularization, resulting in better geometrical structure and coherency of generated objects. Compared to prior work, we achieve more coherent results with higher memory efficiency and faster training speeds.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 37 of its 41 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

37 of 41 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  11. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
25 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  2. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  3. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  4. 2020Neural Sparse Voxel Fieldscited by 959
  5. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  6. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  7. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  8. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  9. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  10. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 510
  11. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 465
  12. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  13. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  14. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  15. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  16. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  17. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  18. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  19. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  20. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  21. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  22. 2022ReLU Fields: The Little Non-linearity That Couldcited by 87
  23. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  24. 2022Towards Implicit Text-Guided 3D Shape Generationcited by 68
  25. 2022CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshescited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 29; 2024: 19; 2025: 4; 2026: 2.

56 papers.

Sort:

By area:

  1. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  2. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  3. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  4. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  5. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  6. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  7. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  8. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generationcited by 181
  9. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  10. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151
  11. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  12. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 122
38 more
  1. 2023Animate124: Animating One Image to 4D Dynamic Scenecited by 113
  2. 2023AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Controlcited by 84
  3. 2024Advances in 3D Generation: A Surveycited by 62
  4. 2024SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfercited by 61
  5. 2023CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layoutcited by 57
  6. 2024MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Modelcited by 55
  7. 2023Instant3D: Instant Text-to-3D Generationcited by 45
  8. 2023Text2Immersion: Generative Immersive Scene with 3D Gaussianscited by 43
  9. 2023Points-to-3D: Bridging the Gap between Sparse Points and Shape-Controllable Text-to-3D Generationcited by 39
  10. 2023X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidancecited by 37
  11. 2024SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Viewscited by 35
  12. 20233D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillationcited by 34
  13. 2023TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervisioncited by 28
  14. 2023Blending-NeRF: Text-Driven Localized Editing in Neural Radiance Fieldscited by 27
  15. 2024InterFusion: Text-Driven Generation of 3D Human-Object Interactioncited by 18
  16. 2024DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraftcited by 17
  17. 2023InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fieldscited by 16
  18. 2024BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Streamcited by 15
  19. 2024Text-to-3D Shape Generationcited by 13
  20. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  21. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  22. 2024DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priorscited by 8
  23. 2023Spice-E : Structural Priors in 3D Diffusion using Cross-Entity Attentioncited by 7
  24. 2024Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priorscited by 7
  25. 2024DreamCraft3D++: Efficient Hierarchical 3D Generation with Multi-Plane Reconstruction Modelcited by 7
  26. 2023Articulated 3D Head Avatar Generation using Text-to-Image Diffusion Modelscited by 6
  27. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  28. 2025Is Single-View Mesh Reconstruction Ready for Robotics?cited by 6
  29. 2024Magic3DSketch: Create Colorful 3D Models From Sketch-Based 3D Modeling Guided by Text and Language-Image Pre-Trainingcited by 5
  30. 2023AlteredAvatar: Stylizing Dynamic 3D Avatars with Fast Style Adaptationcited by 4
  31. 2024A3D: Does Diffusion Dream about 3D Alignment?cited by 3
  32. 2025BloomScene: Lightweight Structured 3D Gaussian Splatting for Crossmodal Scene Generationcited by 3
  33. 2023EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priorcited by 2
  34. 2024iControl3D: An Interactive System for Controllable 3D Scene Generationcited by 2
  35. 2023Antagonising explanation and revealing bias directly through sequencing and multimodal inferencecited by 1
  36. 2023DreamDrone: Text-to-Image Diffusion Models are Zero-shot Perpetual View Generatorscited by 1
  37. 2023Text-Image Conditioned Diffusion for Consistent Text-to-3D Generationcited by 1
  38. 2024HOTS3D: Hyper-Spherical Optimal Transport for Semantic Alignment of Text-to-3D Generationcited by 1

50 most cited of 56 All 56 (JSON)