# Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model

arXiv: https://arxiv.org/abs/2306.09551 · HTML: https://mlsanity.com/paper/2306.09551/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [CLIP-Mesh: Generating textured meshes from text using pretrained image-text models](https://mlsanity.com/paper/2203.13333/) (2022)
- [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021)
- [Editing Conditional Radiance Fields](https://mlsanity.com/paper/2105.06466/) (2021)
- [NeRF-Editing: Geometry Editing of Neural Radiance Fields](https://mlsanity.com/paper/2205.04978/) (2022)
- [SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction](https://mlsanity.com/paper/2212.00792/) (2022)
- [DiffRF: Rendering-Guided 3D Radiance Field Diffusion](https://mlsanity.com/paper/2212.01206/) (2022)
- [Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation](https://mlsanity.com/paper/2303.07937/) (2023)
- [HoloDiffusion: Training a 3D Diffusion Model using 2D Images](https://mlsanity.com/paper/2303.16509/) (2023)
- [Unsupervised Discovery of Object Radiance Fields](https://mlsanity.com/paper/2107.07905/) (2021)
- [Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models](https://mlsanity.com/paper/2212.12990/) (2022)
- [Deformation-Aware 3D Model Embedding and Retrieval](https://mlsanity.com/paper/2004.01228/) (2020)
- [Exploring Compositional Visual Generation with Latent Classifier Guidance](https://mlsanity.com/paper/2304.12536/) (2023)

## Built on by (most cited first, 20 of 20)

- [DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing](https://mlsanity.com/paper/2404.18929/) (2024)
- [View-Consistent 3D Editing with Gaussian Splatting](https://mlsanity.com/paper/2403.11868/) (2024)
- [Alchemist: Parametric Control of Material Properties with Diffusion Models](https://mlsanity.com/paper/2312.02970/) (2023)
- [SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds](https://mlsanity.com/paper/2312.09246/) (2023)
- [GeoDiffuser: Geometry-Based Image Editing with Diffusion Models](https://mlsanity.com/paper/2404.14403/) (2024)
- [SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesis](https://mlsanity.com/paper/2411.16443/) (2024)
- [Learn to Optimize Denoising Scores for 3D Generation: A Unified and Improved Diffusion Prior on NeRF and 3D Gaussian Splatting](https://mlsanity.com/paper/2312.04820/) (2023)
- [ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editing](https://mlsanity.com/paper/2406.09404/) (2024)
- [Applications of Large Scale Foundation Models for Autonomous Driving](https://mlsanity.com/paper/2311.12144/) (2023)
- [AI-Generated Images as Data Source: The Dawn of Synthetic Era](https://mlsanity.com/paper/2310.01830/) (2023)
- [ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing](https://mlsanity.com/paper/2411.05006/) (2024)
- [Fast Multi-view Consistent 3D Editing with Video Priors](https://mlsanity.com/paper/2511.23172/) (2025)
- [Diffusion-Based Attention Warping for Consistent 3D Scene Editing](https://mlsanity.com/paper/2412.07984/) (2024)
- [DATENeRF: Depth-Aware Text-based Editing of NeRFs](https://mlsanity.com/paper/2404.04526/) (2024)
- [NeRF-Insert: 3D Local Editing with Multimodal Control Signals](https://mlsanity.com/paper/2404.19204/) (2024)
- [Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editing](https://mlsanity.com/paper/2603.03143/) (2026)
- [DisCo3D: Distilling Multi-View Consistency for 3D Scene Editing](https://mlsanity.com/paper/2508.01684/) (2025)
- [Personalize Your Gaussian: Consistent 3D Scene Personalization from a Single Image](https://mlsanity.com/paper/2505.14537/) (2025)
- [Diffusion Controller: Framework, Algorithms and Parameterization](https://mlsanity.com/paper/2603.06981/) (2026)
- [InStyle: Instant Appearance Stylization of 3D Shapes](https://mlsanity.com/paper/2512.03683/) (2025)

## Abstract (arXiv)

Recent research has demonstrated that the combination of pretrained diffusion models with neural radiance fields (NeRFs) has emerged as a promising approach for text-to-3D generation. Simply coupling NeRF with diffusion models will result in cross-view inconsistency and degradation of stylized view syntheses. To address this challenge, we propose the Edit-DiffNeRF framework, which is composed of a frozen diffusion model, a proposed delta module to edit the latent semantic space of the diffusion model, and a NeRF. Instead of training the entire diffusion for each scene, our method focuses on editing the latent semantic space in frozen pretrained diffusion models by the delta module. This fundamental change to the standard diffusion framework enables us to make fine-grained modifications to the rendered views and effectively consolidate these instructions in a 3D scene via NeRF training. As a result, we are able to produce an edited 3D scene that faithfully aligns to input text instructions. Furthermore, to ensure semantic consistency across different viewpoints, we propose a novel multi-view semantic consistency loss that extracts a latent semantic embedding from the input view as a prior, and aim to reconstruct it in different views. Our proposed method has been shown to effectively edit real-world 3D scenes, resulting in 25% improvement in the alignment of the performed 3D edits with text instructions compared to prior work.