# SAID-NeRF: Segmentation-AIDed NeRF for Depth Completion of Transparent Objects

arXiv: https://arxiv.org/abs/2403.19607 · HTML: https://mlsanity.com/paper/2403.19607/

## Key facts

- Cited by 5 papers in the map (data through 7 Oct 2026).
- Builds on 40 of its 46 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Builds on (most cited first, 30 of 40)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Dense Depth Priors for Neural Radiance Fields from Sparse Input Views](https://mlsanity.com/paper/2112.03288/) (2021)
- [SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis](https://mlsanity.com/paper/2303.16196/) (2023)
- [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [DeRF: Decomposed Radiance Fields](https://mlsanity.com/paper/2011.12490/) (2020)
- [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022)
- [SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields](https://mlsanity.com/paper/2211.12254/) (2022)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects](https://mlsanity.com/paper/2110.14217/) (2021)
- [NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes](https://mlsanity.com/paper/2111.13260/) (2021)
- [Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement](https://mlsanity.com/paper/2303.02091/) (2023)
- [Segmenting Transparent Objects in the Wild](https://mlsanity.com/paper/2003.13948/) (2020)
- [Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter](https://mlsanity.com/paper/2101.01132/) (2021)
- [NerfAcc: Efficient Sampling Accelerates NeRFs](https://mlsanity.com/paper/2305.04966/) (2023)
- [Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study](https://mlsanity.com/paper/1404.3785/) (2014)
- [GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF](https://mlsanity.com/paper/2210.06575/) (2022)

## Built on by (most cited first, 5 of 5)

- [TranSplat: Surface Embedding-guided 3D Gaussian Splatting for Transparent Object Manipulation](https://mlsanity.com/paper/2502.07840/) (2025)
- [TRAN-D: 2D Gaussian Splatting-based Sparse-view Transparent Object Depth Reconstruction via Physics Simulation for Scene Update](https://mlsanity.com/paper/2507.11069/) (2025)
- [SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Grasping](https://mlsanity.com/paper/2505.24305/) (2025)
- [VAIR: Visuo-Acoustic Implicit Representations for Low-Cost, Multi-Modal Transparent Surface Reconstruction in Indoor Scenes](https://mlsanity.com/paper/2411.04963/) (2024)
- [AISPO: Enhancing Depth Reliability for Robotic Manipulation of Non-Lambertian Objects via Affine-Invariant Shape Prior](https://mlsanity.com/paper/2606.25503/) (2026)

## Abstract (arXiv)

Acquiring accurate depth information of transparent objects using off-the-shelf RGB-D cameras is a well-known challenge in Computer Vision and Robotics. Depth estimation/completion methods are typically employed and trained on datasets with quality depth labels acquired from either simulation, additional sensors or specialized data collection setups and known 3d models. However, acquiring reliable depth information for datasets at scale is not straightforward, limiting training scalability and generalization. Neural Radiance Fields (NeRFs) are learning-free approaches and have demonstrated wide success in novel view synthesis and shape recovery. However, heuristics and controlled environments (lights, backgrounds, etc) are often required to accurately capture specular surfaces. In this paper, we propose using Visual Foundation Models (VFMs) for segmentation in a zero-shot, label-free way to guide the NeRF reconstruction process for these objects via the simultaneous reconstruction of semantic fields and extensions to increase robustness. Our proposed method Segmentation-AIDed NeRF (SAID-NeRF) shows significant performance on depth completion datasets for transparent objects and robotic grasping.