mlsanityData to 7 Oct 2026

Reference-guided Controllable Inpainting of Neural Radiance Fields

arXiv 2304.09677 ↗First posted 19 Apr 2023cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

The popularity of Neural Radiance Fields (NeRFs) for view synthesis has led to a desire for NeRF editing tools. Here, we focus on inpainting regions in a view-consistent and controllable manner. In addition to the typical NeRF inputs and masks delineating the unwanted region in each view, we require only a single inpainted view of the scene, i.e., a reference view. We use monocular depth estimators to back-project the inpainted view to the correct 3D positions. Then, via a novel rendering technique, a bilateral solver can construct view-dependent effects in non-reference views, making the inpainted region appear consistent from any view. For non-reference disoccluded regions, which cannot be supervised by the single reference view, we devise a method based on image inpainters to guide both the geometry and appearance. Our approach shows superior performance to NeRF inpainting baselines, with the additional advantage that a user can control the generated scene via a single inpainted image. Project page: https://ashmrz.github.io/reference-guided-3d

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 71 of its 90 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

71 of 90 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  11. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  12. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
38 more
  1. 2021Vision Transformers for Dense Predictioncited by 1,789
  2. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  3. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  4. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  5. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  6. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  7. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  8. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  9. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
  10. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  11. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 732
  12. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  13. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  14. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  15. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  16. 2018Generative Image Inpainting with Contextual Attentioncited by 578
  17. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  18. 2017Convolutional Sequence to Sequence Learningcited by 550
  19. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  20. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  21. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 465
  22. 2017Wasserstein Auto-Encoderscited by 371
  23. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  24. 2021Advances in Neural Renderingcited by 365
  25. 2022MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architecturescited by 308
  26. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  27. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  28. 2021Editing Conditional Radiance Fieldscited by 222
  29. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 219
  30. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 216
  31. 2022NeRF-Editing: Geometry Editing of Neural Radiance Fieldscited by 216
  32. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 215
  33. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  34. 2023Image Restoration with Mean-Reverting Stochastic Differential Equationscited by 182
  35. 2017Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistencycited by 177
  36. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  37. 2019Pluralistic Image Completioncited by 162
  38. 2021NeRF--: Neural Radiance Fields Without Known Camera Parameterscited by 155

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 16; 2025: 14; 2026: 1.

40 papers.

Sort:

By area:

  1. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  2. 2023NeRFiller: Completing Scenes via Generative 3D Inpaintingcited by 62
  3. 2024InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Priorcited by 55
  4. 2023Watch Your Steps: Local Image and Scene Editing by Text Instructionscited by 50
  5. 2024GScream: Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removalcited by 39
  6. 2024MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editingcited by 30
  7. 2023Point'n Move: Interactive Scene Object Manipulation on Gaussian Splatting Radiance Fieldscited by 27
  8. 2024MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Priorcited by 24
  9. 2024FastScene: Text-Driven Fast 3D Indoor Scene Generation via Panoramic Gaussian Splattingcited by 24
  10. 20253D Gaussian Inpainting with Depth-Guided Cross-View Consistencycited by 23
  11. 2025AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360deg Unbounded Scene Inpaintingcited by 21
  12. 2024Taming Latent Diffusion Model for Neural Radiance Field Inpaintingcited by 20
28 more
  1. 2023InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fieldscited by 16
  2. 2023Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusioncited by 13
  3. 2024ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields with Diffusion Modelscited by 12
  4. 2023LucidDreaming: Controllable Object-Centric 3D Generationcited by 10
  5. 2024InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenescited by 10
  6. 2025IMFine: 3D Inpainting via Geometry-guided Multi-view Refinementcited by 10
  7. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  8. 2025InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Modelcited by 9
  9. 2024ReplaceAnything3D:Text-Guided 3D Scene Editing with Compositional Neural Radiance Fieldscited by 7
  10. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  11. 2024IE-NeRF: Inpainting Enhanced Neural Radiance Fields in the Wildcited by 6
  12. 2025Geometry-Aware Diffusion Models for Multiview Scene Inpaintingcited by 6
  13. 2025DreamInsert: Zero-Shot Image-to-Video Object Insertion from A Single Imagecited by 6
  14. 2023Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representationscited by 5
  15. 2026OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understandingcited by 5
  16. 20234D-Editor: Interactive Object-level Editing in Dynamic Neural Radiance Fields via Semantic Distillationcited by 4
  17. 2024GO-NeRF: Generating Objects in Neural Radiance Fields for Virtual Reality Content Creationcited by 3
  18. 2025Uncertainty-Aware Diffusion Guided Refinement of 3D Scenescited by 3
  19. 2025DiGA3D: Coarse-to-Fine Diffusional Propagation of Geometry and Appearance for Versatile 3D Inpaintingcited by 3
  20. 2025High-fidelity 3D Gaussian Inpainting: preserving multi-view consistency and photorealistic detailscited by 3
  21. 2025Fillerbuster: Unified Generative Scene Completion Model for Casual Capturescited by 2
  22. 2025SplatFill: 3D Scene Inpainting via Depth-Guided Gaussian Splattingcited by 2
  23. 2024SR-CurvANN: Advancing 3D Surface Reconstruction through Curvature-Aware Neural Networkscited by 1
  24. 2024Towards High-Fidelity 3D Portrait Generation with Rich Details by Cross-View Prior-Aware Diffusioncited by 1
  25. 2025NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embeddingcited by 1
  26. 2023Survey on Controlable Image Synthesis with Deep Learningcited by 0
  27. 2025Defurnishing with X-Ray Vision: Joint Removal of Furniture from Panoramas and Meshcited by 0
  28. 2025RePaintGS: Reference-Guided Gaussian Splatting for Realistic and View-Consistent 3D Scene Inpaintingcited by 0