# Open-NeRF: Towards Open Vocabulary NeRF Decomposition arXiv: https://arxiv.org/abs/2310.16383 ยท HTML: https://mlsanity.com/paper/2310.16383/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 40 of its 48 references that are in the map. - First posted on arXiv 25 Oct 2023. ## Builds on (most cited first, 30 of 40) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017) - [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021) - [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021) - [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021) - [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021) - [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020) - [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020) - [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021) - [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022) - [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022) - [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023) - [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023) - [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022) - [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021) - [Dynamic View Synthesis from Dynamic Monocular Video](https://mlsanity.com/paper/2105.06468/) (2021) - [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021) - [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022) - [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021) - [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021) - [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022) - [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023) - [Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations](https://mlsanity.com/paper/2209.03494/) (2022) - [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023) - [Panoptic Lifting for 3D Scene Understanding with Neural Fields](https://mlsanity.com/paper/2212.09802/) (2022) ## Built on by (most cited first, 13 of 13) - [Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting](https://mlsanity.com/paper/2503.14029/) (2025) - [GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting](https://mlsanity.com/paper/2411.07555/) (2024) - [Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space](https://mlsanity.com/paper/2408.07416/) (2024) - [GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields](https://mlsanity.com/paper/2404.00931/) (2024) - [PhysRig: Differentiable Physics-Based Skinning and Rigging Framework for Realistic Articulated Object Modeling](https://mlsanity.com/paper/2506.20936/) (2025) - [Improving Multi-label Recognition using Class Co-Occurrence Probabilities](https://mlsanity.com/paper/2404.16193/) (2024) - [LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control](https://mlsanity.com/paper/2406.16038/) (2024) - [Self-Calibrating 4D Novel View Synthesis from Monocular Videos Using Gaussian Splatting](https://mlsanity.com/paper/2406.01042/) (2024) - [Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation](https://mlsanity.com/paper/2512.22745/) (2025) - [RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregation](https://mlsanity.com/paper/2505.15373/) (2025) - [SERES: Semantic-aware neural reconstruction from sparse views](https://mlsanity.com/paper/2508.18314/) (2025) - [PinPoint3D: Fine-Grained 3D Part Segmentation from a Few Clicks](https://mlsanity.com/paper/2509.25970/) (2025) - [CoRef-GS: Cooperative Referring Gaussian Splatting for Multi-Agent Scene Understanding](https://mlsanity.com/paper/2609.20586/) (2026) ## Abstract (arXiv) In this paper, we address the challenge of decomposing Neural Radiance Fields (NeRF) into objects from an open vocabulary, a critical task for object manipulation in 3D reconstruction and view synthesis. Current techniques for NeRF decomposition involve a trade-off between the flexibility of processing open-vocabulary queries and the accuracy of 3D segmentation. We present, Open-vocabulary Embedded Neural Radiance Fields (Open-NeRF), that leverage large-scale, off-the-shelf, segmentation models like the Segment Anything Model (SAM) and introduce an integrate-and-distill paradigm with hierarchical embeddings to achieve both the flexibility of open-vocabulary querying and 3D segmentation accuracy. Open-NeRF first utilizes large-scale foundation models to generate hierarchical 2D mask proposals from varying viewpoints. These proposals are then aligned via tracking approaches and integrated within the 3D space and subsequently distilled into the 3D field. This process ensures consistent recognition and granularity of objects from different viewpoints, even in challenging scenarios involving occlusion and indistinct features. Our experimental results show that the proposed Open-NeRF outperforms state-of-the-art methods such as LERF citelerf and FFD citeffd in open-vocabulary scenarios. Open-NeRF offers a promising solution to NeRF decomposition, guided by open-vocabulary queries, enabling novel applications in robotics and vision-language interaction in open-world 3D scenes.