mlsanityData to 14 Oct 2026

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

arXiv 2112.05139 ↗First posted 9 Dec 2021cs.CV cs.GRcited by 370

Not on the map. Why?

Cited by 53 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present CLIP-NeRF, a multi-modal 3D object manipulation method for neural radiance fields (NeRF). By leveraging the joint language-image embedding space of the recent Contrastive Language-Image Pre-Training (CLIP) model, we propose a unified framework that allows manipulating NeRF in a user-friendly way, using either a short text prompt or an exemplar image. Specifically, to combine the novel view synthesis capability of NeRF and the controllable manipulation ability of latent representations from generative models, we introduce a disentangled conditional NeRF architecture that allows individual control over both shape and appearance. This is achieved by performing the shape conditioning via applying a learned deformation field to the positional encoding and deferring color conditioning to the volumetric rendering stage. To bridge this disentangled latent representation to the CLIP embedding, we design two code mappers that take a CLIP embedding as input and update the latent codes to reflect the targeted editing. The mappers are trained with a CLIP-based matching loss to ensure the manipulation accuracy. Furthermore, we propose an inverse optimization method that accurately projects an input image to the latent codes for manipulation to enable editing on real images. We evaluate our approach by extensive experiments on a variety of text prompts and exemplar images and also provide an intuitive interface for interactive editing. Our implementation is available at https://cassiepython.github.io/clipnerf/

Key facts. Cited by 370 papers in the map (data through 14 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 9 Dec 2021.

Builds on

33 of 45 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  7. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  8. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  9. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  10. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  11. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  12. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
21 more
  1. 2018Which Training Methods for GANs do actually Converge?cited by 679
  2. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  3. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 496
  4. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 471
  5. 2020Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstructioncited by 467
  6. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  7. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 449
  8. 2020Local Implicit Grid Representations for 3D Scenescited by 373
  9. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  10. 2020Free View Synthesiscited by 231
  11. 2020Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulationcited by 225
  12. 2021Editing Conditional Radiance Fieldscited by 222
  13. 2017Learning a Multi-View Stereo Machinecited by 194
  14. 2019Image Processing Using Multi-Code GAN Priorcited by 185
  15. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  16. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  17. 2019Text-based Editing of Talking-head Videocited by 103
  18. 20193DN: 3D Deformation Networkcited by 60
  19. 2020Deformation-Aware 3D Model Embedding and Retrievalcited by 30
  20. 2020Learning Implicit Surface Light Fieldscited by 23
  21. 2018PhotoShape: Photorealistic Materials for Large-Scale Shape Collectionscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 60; 2023: 132; 2024: 104; 2025: 52; 2026: 21.

370 papers.

Sort:

By area:

  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  2. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 587
  3. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  4. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  5. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 408
  6. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 393
  7. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  8. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 335
  9. 2023Text-to-3D using Gaussian Splattingcited by 299
  10. 2023FSGS: Real-Time Few-shot View Synthesis using Gaussian Splattingcited by 279
  11. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  12. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
38 more
  1. 2023Text-To-4D Dynamic Scene Generationcited by 205
  2. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  3. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  4. 2023GaussianEditor: Editing 3D Gaussians Delicately with Text Instructionscited by 168
  5. 2022SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Imagecited by 165
  6. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 158
  7. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  8. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 155
  9. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  10. 2022Diffusion-based Image Translation using Disentangled Style and Content Representationcited by 149
  11. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Viewscited by 148
  12. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  13. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  14. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 118
  15. 2023Vox-E: Text-guided Voxel Editing of 3D Objectscited by 115
  16. 2023TADA! Text to Animatable Digital Avatarscited by 110
  17. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  18. 2023Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinementcited by 106
  19. 2023SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Fieldcited by 94
  20. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 94
  21. 2022Compressible-composable NeRF via Rank-residual Decompositioncited by 92
  22. 2023HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splattingcited by 89
  23. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 89
  24. 2022TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decompositioncited by 83
  25. 2022BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fieldscited by 80
  26. 2024StyleGaussian: Instant 3D Style Transfer with Gaussian Splattingcited by 79
  27. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 78
  28. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  29. 2024TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Promptscited by 77
  30. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 77
  31. 2023SKED: Sketch-guided Text-based 3D Editingcited by 73
  32. 2022NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenescited by 71
  33. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  34. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 69
  35. 20223D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Modelscited by 67
  36. 2023Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priorscited by 64
  37. 2023GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learningcited by 63
  38. 2024Advances in 3D Generation: A Surveycited by 63

50 most cited of 370 All 370 (JSON)