# NeTO:Neural Reconstruction of Transparent Objects with Self-Occlusion Aware Refraction-Tracing

arXiv: https://arxiv.org/abs/2303.11219 · HTML: https://mlsanity.com/paper/2303.11219/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 65 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 29 of 29)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views](https://mlsanity.com/paper/2206.05737/) (2022)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Neural 3D Scene Reconstruction with the Manhattan-world Assumption](https://mlsanity.com/paper/2205.02836/) (2022)
- [Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control](https://mlsanity.com/paper/2106.02019/) (2021)
- [Neural 3D Reconstruction in the Wild](https://mlsanity.com/paper/2205.12955/) (2022)
- [NeuralUDF: Learning Unsigned Distance Fields for Multi-view Reconstruction of Surfaces with Arbitrary Topologies](https://mlsanity.com/paper/2211.14173/) (2022)
- [TOM-Net: Learning Transparent Object Matting from a Single Image](https://mlsanity.com/paper/1803.04636/) (2018)
- [Through the Looking Glass: Neural 3D Reconstruction of Transparent Shapes](https://mlsanity.com/paper/2004.10904/) (2020)
- [Eikonal Fields for Refractive Novel-View Synthesis](https://mlsanity.com/paper/2202.00948/) (2022)
- [Full 3D Reconstruction of Transparent Objects](https://mlsanity.com/paper/1805.03482/) (2018)
- [Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects](https://mlsanity.com/paper/2009.09144/) (2020)
- [Hybrid Mesh-neural Representation for 3D Transparent Object Reconstruction](https://mlsanity.com/paper/2203.12613/) (2022)
- [Polarimetric Inverse Rendering for Transparent Shapes Reconstruction](https://mlsanity.com/paper/2208.11836/) (2022)

## Built on by (most cited first, 9 of 9)

- [NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields](https://mlsanity.com/paper/2309.13039/) (2023)
- [TransparentGS: Fast Inverse Rendering of Transparent Objects with Gaussians](https://mlsanity.com/paper/2504.18768/) (2025)
- [ClearDepth: Enhanced Stereo Perception of Transparent Objects for Robotic Manipulation](https://mlsanity.com/paper/2409.08926/) (2024)
- [Interactive Occlusion Boundary Estimation through Exploitation of Synthetic Data](https://mlsanity.com/paper/2408.15038/) (2024)
- [From Transparent to Opaque: Rethinking Neural Implicit Surfaces with α-NeuS](https://mlsanity.com/paper/2411.05362/) (2024)
- [TransNormal: Dense Visual Semantics for Diffusion-based Transparent Object Normal Estimation](https://mlsanity.com/paper/2602.00839/) (2026)
- [RefRef: A Dataset and Benchmark for Reconstructing Refractive and Reflective Objects](https://mlsanity.com/paper/2505.05848/) (2025)
- [DiffTrans: Differentiable Geometry-Materials Decomposition for Reconstructing Transparent Objects](https://mlsanity.com/paper/2603.00413/) (2026)
- [TransNormal-2: Geometry-Grounded Rectified Flow with Edge-Aware Decoding for Precise Normal Estimation](https://mlsanity.com/paper/2609.06665/) (2026)

## Abstract (arXiv)

We present a novel method, called NeTO, for capturing 3D geometry of solid transparent objects from 2D images via volume rendering. Reconstructing transparent objects is a very challenging task, which is ill-suited for general-purpose reconstruction techniques due to the specular light transport phenomena. Although existing refraction-tracing based methods, designed specially for this task, achieve impressive results, they still suffer from unstable optimization and loss of fine details, since the explicit surface representation they adopted is difficult to be optimized, and the self-occlusion problem is ignored for refraction-tracing. In this paper, we propose to leverage implicit Signed Distance Function (SDF) as surface representation, and optimize the SDF field via volume rendering with a self-occlusion aware refractive ray tracing. The implicit representation enables our method to be capable of reconstructing high-quality reconstruction even with a limited set of images, and the self-occlusion aware strategy makes it possible for our method to accurately reconstruct the self-occluded regions. Experiments show that our method achieves faithful reconstruction results and outperforms prior works by a large margin. Visit our project page at https://www.xxlong.site/NeTO/