# LaTeRF: Label and Text Driven Object Radiance Fields

arXiv: https://arxiv.org/abs/2207.01583 · HTML: https://mlsanity.com/paper/2207.01583/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 56 references that are in the map.
- First posted on arXiv 4 Jul 2022.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](https://mlsanity.com/paper/2104.00677/) (2021)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes](https://mlsanity.com/paper/2101.10994/) (2021)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)

## Built on by (most cited first, 30 of 38)

- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields](https://mlsanity.com/paper/2211.12254/) (2022)
- [SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field](https://mlsanity.com/paper/2303.13277/) (2023)
- [SKED: Sketch-guided Text-based 3D Editing](https://mlsanity.com/paper/2303.10735/) (2023)
- [Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion](https://mlsanity.com/paper/2306.04633/) (2023)
- [CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layout](https://mlsanity.com/paper/2303.13843/) (2023)
- [InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior](https://mlsanity.com/paper/2404.11613/) (2024)
- [Watch Your Steps: Local Image and Scene Editing by Text Instructions](https://mlsanity.com/paper/2308.08947/) (2023)
- [EPIC Fields: Marrying 3D Geometry and Video Understanding](https://mlsanity.com/paper/2306.08731/) (2023)
- [Reference-guided Controllable Inpainting of Neural Radiance Fields](https://mlsanity.com/paper/2304.09677/) (2023)
- [One-shot Implicit Animatable Avatars with Model-based Priors](https://mlsanity.com/paper/2212.02469/) (2022)
- [OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fields](https://mlsanity.com/paper/2305.10503/) (2023)
- [DepthLab: From Partial to Complete](https://mlsanity.com/paper/2412.18153/) (2024)
- [MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior](https://mlsanity.com/paper/2405.02859/) (2024)
- [RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models](https://mlsanity.com/paper/2306.05668/) (2023)
- [A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation](https://mlsanity.com/paper/2508.09977/) (2025)
- [InterFusion: Text-Driven Generation of 3D Human-Object Interaction](https://mlsanity.com/paper/2403.15612/) (2024)
- [InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fields](https://mlsanity.com/paper/2305.15094/) (2023)
- [Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models](https://mlsanity.com/paper/2312.04533/) (2023)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [AutoRecon: Automated 3D Object Discovery and Reconstruction](https://mlsanity.com/paper/2305.08810/) (2023)
- [Neural Radiance Fields for the Real World: A Survey](https://mlsanity.com/paper/2501.13104/) (2025)
- [LucidDreaming: Controllable Object-Centric 3D Generation](https://mlsanity.com/paper/2312.00588/) (2023)
- [Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Review](https://mlsanity.com/paper/2402.11141/) (2024)
- [View-consistent Object Removal in Radiance Fields](https://mlsanity.com/paper/2408.02100/) (2024)
- [Planar Gaussian Splatting](https://mlsanity.com/paper/2412.01931/) (2024)
- [WildSeg3D: Segment Any 3D Objects in the Wild from 2D Images](https://mlsanity.com/paper/2503.08407/) (2025)
- [Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations](https://mlsanity.com/paper/2310.17880/) (2023)
- [ClusteringSDF: Self-Organized Neural Implicit Surfaces for 3D Decomposition](https://mlsanity.com/paper/2403.14619/) (2024)
- [Remove360: Benchmarking Residuals After Object Removal in 3D Gaussian Splatting](https://mlsanity.com/paper/2508.11431/) (2025)

## Abstract (arXiv)

Obtaining 3D object representations is important for creating photo-realistic simulations and for collecting AR and VR assets. Neural fields have shown their effectiveness in learning a continuous volumetric representation of a scene from 2D images, but acquiring object representations from these models with weak supervision remains an open challenge. In this paper we introduce LaTeRF, a method for extracting an object of interest from a scene given 2D images of the entire scene, known camera poses, a natural language description of the object, and a set of point-labels of object and non-object points in the input images. To faithfully extract the object from the scene, LaTeRF extends the NeRF formulation with an additional `objectness' probability at each 3D point. Additionally, we leverage the rich latent space of a pre-trained CLIP model combined with our differentiable object renderer, to inpaint the occluded parts of the object. We demonstrate high-fidelity object extraction on both synthetic and real-world datasets and justify our design choices through an extensive ablation study.