# Hybrid Mesh-neural Representation for 3D Transparent Object Reconstruction

arXiv: https://arxiv.org/abs/2203.12613 · HTML: https://mlsanity.com/paper/2203.12613/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 75 references that are in the map.
- First posted on arXiv 23 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [MeshSDF: Differentiable Iso-Surface Extraction](https://mlsanity.com/paper/2006.03997/) (2020)
- [TOM-Net: Learning Transparent Object Matting from a Single Image](https://mlsanity.com/paper/1803.04636/) (2018)
- [Through the Looking Glass: Neural 3D Reconstruction of Transparent Shapes](https://mlsanity.com/paper/2004.10904/) (2020)
- [Full 3D Reconstruction of Transparent Objects](https://mlsanity.com/paper/1805.03482/) (2018)
- [Deep Surface Light Fields](https://mlsanity.com/paper/1810.06514/) (2018)
- [Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects](https://mlsanity.com/paper/2009.09144/) (2020)
- [Differentiable Visual Computing](https://mlsanity.com/paper/1904.12228/) (2019)

## Built on by (most cited first, 15 of 15)

- [GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs](https://mlsanity.com/paper/2312.00093/) (2023)
- [Learning Disentangled Avatars with Hybrid 3D Representations](https://mlsanity.com/paper/2309.06441/) (2023)
- [Looking Through the Glass: Neural Surface Reconstruction Against High Specular Reflections](https://mlsanity.com/paper/2304.08706/) (2023)
- [SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild](https://mlsanity.com/paper/2401.10171/) (2024)
- [NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields](https://mlsanity.com/paper/2309.13039/) (2023)
- [TransparentGS: Fast Inverse Rendering of Transparent Objects with Gaussians](https://mlsanity.com/paper/2504.18768/) (2025)
- [NEMTO: Neural Environment Matting for Novel View and Relighting Synthesis of Transparent Objects](https://mlsanity.com/paper/2303.11963/) (2023)
- [NeTO:Neural Reconstruction of Transparent Objects with Self-Occlusion Aware Refraction-Tracing](https://mlsanity.com/paper/2303.11219/) (2023)
- [Polarimetric Inverse Rendering for Transparent Shapes Reconstruction](https://mlsanity.com/paper/2208.11836/) (2022)
- [SViM3D: Stable Video Material Diffusion for Single Image 3D Generation](https://mlsanity.com/paper/2510.08271/) (2025)
- [RFTrans: Leveraging Refractive Flow of Transparent Objects for Surface Normal Estimation and Manipulation](https://mlsanity.com/paper/2311.12398/) (2023)
- [Sparse 3D Reconstruction via Object-Centric Ray Sampling](https://mlsanity.com/paper/2309.03008/) (2023)
- [LaFiTe: A Generative Latent Field for 3D Native Texturing](https://mlsanity.com/paper/2512.04786/) (2025)
- [Tabletop Transparent Scene Reconstruction via Epipolar-Guided Optical Flow with Monocular Depth Completion Prior](https://mlsanity.com/paper/2310.09956/) (2023)
- [DiffTrans: Differentiable Geometry-Materials Decomposition for Reconstructing Transparent Objects](https://mlsanity.com/paper/2603.00413/) (2026)

## Abstract (arXiv)

We propose a novel method to reconstruct the 3D shapes of transparent objects using hand-held captured images under natural light conditions. It combines the advantage of explicit mesh and multi-layer perceptron (MLP) network, a hybrid representation, to simplify the capture setting used in recent contributions. After obtaining an initial shape through the multi-view silhouettes, we introduce surface-based local MLPs to encode the vertex displacement field (VDF) for the reconstruction of surface details. The design of local MLPs allows to represent the VDF in a piece-wise manner using two layer MLP networks, which is beneficial to the optimization algorithm. Defining local MLPs on the surface instead of the volume also reduces the searching space. Such a hybrid representation enables us to relax the ray-pixel correspondences that represent the light path constraint to our designed ray-cell correspondences, which significantly simplifies the implementation of single-image based environment matting algorithm. We evaluate our representation and reconstruction algorithm on several transparent objects with ground truth models. Our experiments show that our method can produce high-quality reconstruction results superior to state-of-the-art methods using a simplified data acquisition setup.