mlsanityData to 14 Oct 2026

Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions

arXiv 2303.12789 ↗First posted 22 Mar 2023cs.CV cs.GRcited by 484

Not on the map. Why?

Cited by 121 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a method for editing NeRF scenes with text-instructions. Given a NeRF of a scene and the collection of images used to reconstruct it, our method uses an image-conditioned diffusion model (InstructPix2Pix) to iteratively edit the input images while optimizing the underlying scene, resulting in an optimized 3D scene that respects the edit instruction. We demonstrate that our proposed method is able to edit large-scale, real-world scenes, and is able to accomplish more realistic, targeted edits than prior work.

Key facts. Cited by 484 papers in the map (data through 14 Oct 2026). Builds on 44 of its 61 references that are in the map. First posted on arXiv 22 Mar 2023.

Builds on

44 of 61 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
32 more
  1. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  2. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  3. 2022Language-driven Semantic Segmentationcited by 717
  4. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 600
  5. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  6. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 575
  7. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  8. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  9. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 496
  10. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  11. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  12. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 366
  13. 2021Advances in Neural Renderingcited by 365
  14. 2022Visual Language Maps for Robot Navigationcited by 365
  15. 2021NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Imagescited by 336
  16. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 334
  17. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  18. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  19. 2021Editing Conditional Radiance Fieldscited by 222
  20. 2022NeRF-Editing: Geometry Editing of Neural Radiance Fieldscited by 217
  21. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  22. 2021Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decompositioncited by 205
  23. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  24. 2021Editable Free-viewpoint Video Using a Layered Neural Representationcited by 148
  25. 2022ARF: Artistic Radiance Fieldscited by 147
  26. 2022StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learningcited by 140
  27. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  28. 2021Stylizing 3D Scene via Implicit Representation and HyperNetworkcited by 103
  29. 2021Unsupervised Discovery of Object Radiance Fieldscited by 101
  30. 2021Learning to Stylize Novel Viewscited by 76
  31. 2022Understanding Pure CLIP Guidance for Voxel Grid NeRF Modelscited by 56
  32. 2022PaletteNeRF: Palette-based Color Editing for NeRFscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 96; 2024: 173; 2025: 135; 2026: 80.

484 papers.

Sort:

By area:

  1. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  2. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 393
  3. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 335
  4. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 318
  5. 2024FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Modelscited by 217
  6. 2023GaussianEditor: Editing 3D Gaussians Delicately with Text Instructionscited by 168
  7. 2025Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Modelscited by 161
  8. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 155
  9. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 150
  10. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  11. 2024Recent Advances in 3D Gaussian Splattingcited by 131
  12. 2023Text-to-3D with Classifier Score Distillationcited by 128
38 more
  1. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  2. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  3. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 117
  4. 2024GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editingcited by 115
  5. 2023EditGuard: Versatile Image Watermarking for Tamper Localization and Copyright Protectioncited by 111
  6. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidancecited by 110
  7. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  8. 2023CoDeF: Content Deformation Fields for Temporally Consistent Video Processingcited by 108
  9. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 94
  10. 2023HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generationcited by 88
  11. 2024StyleGaussian: Instant 3D Style Transfer with Gaussian Splattingcited by 79
  12. 2024TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Promptscited by 77
  13. 2023FocalDreamer: Text-driven 3D Editing via Focal-fusion Assemblycited by 76
  14. 2023SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusioncited by 72
  15. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 70
  16. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 69
  17. 2024ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fieldscited by 68
  18. 2024View-Consistent 3D Editing with Gaussian Splattingcited by 67
  19. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesiscited by 63
  20. 2024Advances in 3D Generation: A Surveycited by 63
  21. 2023NeRFiller: Completing Scenes via Generative 3D Inpaintingcited by 62
  22. 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 62
  23. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  24. 2024GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splattingcited by 59
  25. 2024Click-Gaussian: Interactive Segmentation to Any 3D Gaussianscited by 58
  26. 2024FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimallycited by 58
  27. 2024ViewDiff: 3D-Consistent Image Generation with Text-to-Image Modelscited by 57
  28. 2024FlashTex: Fast Relightable Mesh Texturing with LightControlNetcited by 56
  29. 2024InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Priorcited by 56
  30. 2024Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collectionscited by 56
  31. 2023RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texturecited by 53
  32. 2023Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Promptscited by 52
  33. 2023PIE-NeRF: Physics-based Interactive Elastodynamics with NeRFcited by 51
  34. 2023Watch Your Steps: Local Image and Scene Editing by Text Instructionscited by 50
  35. 2023HeadSculpt: Crafting 3D Head Avatars with Textcited by 49
  36. 2024Generic 3D Diffusion Adapter Using Controlled Multi-View Editingcited by 47
  37. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  38. 2023Posterior Distillation Samplingcited by 46

50 most cited of 484 All 484 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models
  2. 2024SealD-NeRF: Interactive Pixel-Level Editing for Dynamic Scenes by Neural Radiance Fields
  3. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models
  4. 2023SKED: Sketch-guided Text-based 3D Editing
  5. 2023Pix2Video: Video Editing using Image Diffusion
  6. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them First
  7. 2024Text-to-3D Generation by 2D Editing
  8. 2024LoMOE: Localized Multi-Object Editing via Multi-Diffusion