# RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models

arXiv: https://arxiv.org/abs/2306.05668 · HTML: https://mlsanity.com/paper/2306.05668/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022)
- [Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering](https://mlsanity.com/paper/2109.01847/) (2021)
- [Editing Conditional Radiance Fields](https://mlsanity.com/paper/2105.06466/) (2021)
- [NeRF-Editing: Geometry Editing of Neural Radiance Fields](https://mlsanity.com/paper/2205.04978/) (2022)
- [Deforming Radiance Fields with Cages](https://mlsanity.com/paper/2207.12298/) (2022)
- [Removing Objects From Neural Radiance Fields](https://mlsanity.com/paper/2212.11966/) (2022)
- [AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithms](https://mlsanity.com/paper/1901.07849/) (2019)
- [LaTeRF: Label and Text Driven Object Radiance Fields](https://mlsanity.com/paper/2207.01583/) (2022)

## Built on by (most cited first, 19 of 19)

- [DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing](https://mlsanity.com/paper/2404.18929/) (2024)
- [View-Consistent 3D Editing with Gaussian Splatting](https://mlsanity.com/paper/2403.11868/) (2024)
- [Alchemist: Parametric Control of Material Properties with Diffusion Models](https://mlsanity.com/paper/2312.02970/) (2023)
- [SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds](https://mlsanity.com/paper/2312.09246/) (2023)
- [Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning](https://mlsanity.com/paper/2405.08054/) (2024)
- [MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior](https://mlsanity.com/paper/2405.02859/) (2024)
- [Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training](https://mlsanity.com/paper/2312.01663/) (2023)
- [Fast Multi-view Consistent 3D Editing with Video Priors](https://mlsanity.com/paper/2511.23172/) (2025)
- [InstantStyleGaussian: Efficient Art Style Transfer with 3D Gaussian Splatting](https://mlsanity.com/paper/2408.04249/) (2024)
- [TrAME: Trajectory-Anchored Multi-View Editing for Text-Guided 3D Gaussian Splatting Manipulation](https://mlsanity.com/paper/2407.02034/) (2024)
- [O2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model](https://mlsanity.com/paper/2308.09591/) (2023)
- [GaussianForest: Hierarchical-Hybrid 3D Gaussian Splatting for Compressed Scene Modeling](https://mlsanity.com/paper/2406.08759/) (2024)
- [Stable Score Distillation](https://mlsanity.com/paper/2507.09168/) (2025)
- [Towards a Training Free Approach for 3D Scene Editing](https://mlsanity.com/paper/2412.12766/) (2024)
- [Personalize Your Gaussian: Consistent 3D Scene Personalization from a Single Image](https://mlsanity.com/paper/2505.14537/) (2025)
- [TEDRA: Text-based Editing of Dynamic and Photoreal Actors](https://mlsanity.com/paper/2408.15995/) (2024)
- [Online 3D Gaussian Splatting Modeling with Novel View Selection](https://mlsanity.com/paper/2508.14014/) (2025)
- [CADMorph: Geometry-Driven Parametric CAD Editing via a Plan-Generate-Verify Loop](https://mlsanity.com/paper/2512.11480/) (2025)
- [First Shape, Then Meaning: Efficient Geometry and Semantics Learning for Indoor Reconstruction](https://mlsanity.com/paper/2605.03463/) (2026)

## Abstract (arXiv)

The emergence of Neural Radiance Fields (NeRF) has promoted the development of synthesized high-fidelity views of the intricate real world. However, it is still a very demanding task to repaint the content in NeRF. In this paper, we propose a novel framework that can take RGB images as input and alter the 3D content in neural scenes. Our work leverages existing diffusion models to guide changes in the designated 3D content. Specifically, we semantically select the target object and a pre-trained diffusion model will guide the NeRF model to generate new 3D objects, which can improve the editability, diversity, and application range of NeRF. Experiment results show that our algorithm is effective for editing 3D objects in NeRF under different text prompts, including editing appearance, shape, and more. We validate our method on both real-world datasets and synthetic-world datasets for these editing tasks. Please visit https://starstesla.github.io/repaintnerf for a better view of our results.