mlsanityData to 7 Oct 2026

Decomposing NeRF for Editing via Feature Field Distillation

arXiv 2205.15585 ↗First posted 31 May 2022cs.CV cs.GRcited by 370

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations. However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional. In particular, it has been difficult to selectively edit specific regions or objects. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes. We propose to distill the knowledge of off-the-shelf, self-supervised 2D image feature extractors such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the radiance field. Given a user-specified query of various modalities such as text, an image patch, or a point-and-click selection, 3D feature fields semantically decompose 3D space without the need for re-training and enable us to semantically select and edit regions in the radiance field. Our experiments validate that the distilled feature fields (DFFs) can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.

Key facts. Cited by 370 papers in the map (data through 7 Oct 2026). Builds on 94 of its 131 references that are in the map. First posted on arXiv 31 May 2022.

Builds on

94 of 131 references.

References · 94

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
38 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  3. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  4. 2021Vision Transformers for Dense Predictioncited by 1,789
  5. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  6. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  7. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  8. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  9. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  10. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  11. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  13. 2020Neural Sparse Voxel Fieldscited by 959
  14. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  15. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  16. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  17. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  18. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  19. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
  20. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  21. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  22. 2022Language-driven Semantic Segmentationcited by 711
  23. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  24. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  25. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  26. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  27. 2019Deep Learning for 3D Point Clouds: A Surveycited by 557
  28. 2021Neural Fields in Visual Computing and Beyondcited by 539
  29. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  30. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  31. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  32. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  33. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 446
  34. 2021Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapescited by 426
  35. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  36. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  37. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  38. 2021Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughscited by 396

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 21; 2023: 108; 2024: 130; 2025: 72; 2026: 39.

370 papers.

Sort:

By area:

  1. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 582
  2. 2023LERF: Language Embedded Radiance Fieldscited by 538
  3. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  4. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 420
  5. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  6. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  7. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
  8. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 361
  9. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  10. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  11. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 331
  12. 2023FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularizationcited by 310
38 more
  1. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  2. 2024gsplat: An Open-Source Library for Gaussian Splattingcited by 279
  3. 2023FSGS: Real-Time Few-shot View Synthesis using Gaussian Splattingcited by 278
  4. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  5. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  6. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  7. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  8. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 171
  9. 2023GaussianEditor: Editing 3D Gaussians Delicately with Text Instructionscited by 166
  10. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  11. 2023Segment Any 3D Gaussianscited by 158
  12. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 156
  13. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 154
  14. 2022NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fieldscited by 151
  15. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  16. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  17. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  18. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  19. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  20. 2023Vox-E: Text-guided Voxel Editing of 3D Objectscited by 114
  21. 2024FeatUp: A Model-Agnostic Framework for Features at Any Resolutioncited by 114
  22. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  23. 2024GARField: Group Anything with Radiance Fieldscited by 108
  24. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  25. 2023SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Fieldcited by 93
  26. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 92
  27. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  28. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  29. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  30. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  31. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 81
  32. 2023StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fieldscited by 79
  33. 2024StyleGaussian: Instant 3D Style Transfer with Gaussian Splattingcited by 78
  34. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
  35. 2024TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Promptscited by 76
  36. 2023FocalDreamer: Text-driven 3D Editing via Focal-fusion Assemblycited by 75
  37. 2023OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learningcited by 75
  38. 2024Large Spatial Model: End-to-end Unposed Images to Semantic 3Dcited by 73

50 most cited of 370 All 370 (JSON)