# Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects

arXiv: https://arxiv.org/abs/2110.00087 · HTML: https://mlsanity.com/paper/2110.00087/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 66 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Non-Local Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2007.10042/) (2020)
- [Single-Image Depth Perception in the Wild](https://mlsanity.com/paper/1604.03901/) (2016)
- [GRNet: Gridding Residual Network for Dense Point Cloud Completion](https://mlsanity.com/paper/2006.03761/) (2020)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [High Quality Monocular Depth Estimation via Transfer Learning](https://mlsanity.com/paper/1812.11941/) (2018)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [Learning Guided Convolutional Network for Depth Completion](https://mlsanity.com/paper/1908.01238/) (2019)
- [CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion](https://mlsanity.com/paper/1911.05377/) (2019)
- [S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds](https://mlsanity.com/paper/2012.09242/) (2020)
- [Segmenting Transparent Objects in the Wild](https://mlsanity.com/paper/2003.13948/) (2020)
- [Guiding Monocular Depth Estimation Using Depth-Attention Volume](https://mlsanity.com/paper/2004.02760/) (2020)
- [Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimation](https://mlsanity.com/paper/1803.11029/) (2018)

## Built on by (most cited first, 30 of 39)

- [ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization](https://mlsanity.com/paper/2401.06949/) (2024)
- [TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline](https://mlsanity.com/paper/2202.08471/) (2022)
- [Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects](https://mlsanity.com/paper/2208.03792/) (2022)
- [ClearPose: Large-scale Transparent Object Dataset and Benchmark](https://mlsanity.com/paper/2203.03890/) (2022)
- [StereoPose: Category-Level 6D Transparent Object Pose Estimation from Stereo Images via Back-View NOCS](https://mlsanity.com/paper/2211.01644/) (2022)
- [TRansPose: Large-Scale Multispectral Dataset for Transparent Object](https://mlsanity.com/paper/2307.05016/) (2023)
- [MVTrans: Multi-View Perception of Transparent Objects](https://mlsanity.com/paper/2302.11683/) (2023)
- [TransNet: Category-Level Transparent Object Pose Estimation](https://mlsanity.com/paper/2208.10002/) (2022)
- [FDCT: Fast Depth Completion for Transparent Objects](https://mlsanity.com/paper/2307.12274/) (2023)
- [LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow](https://mlsanity.com/paper/2409.05688/) (2024)
- [ClearDepth: Enhanced Stereo Perception of Transparent Objects for Robotic Manipulation](https://mlsanity.com/paper/2409.08926/) (2024)
- [Diffusion-Based Depth Inpainting for Transparent and Reflective Objects](https://mlsanity.com/paper/2410.08567/) (2024)
- [Monocular Depth Estimation and Segmentation for Transparent Object with Iterative Semantic and Geometric Fusion](https://mlsanity.com/paper/2502.14616/) (2025)
- [Seeing and Seeing Through the Glass: Real and Synthetic Data for Multi-Layer Depth Estimation](https://mlsanity.com/paper/2503.11633/) (2025)
- [TransDiff: Diffusion-Based Method for Manipulating Transparent Objects Using a Single RGB-D Image](https://mlsanity.com/paper/2503.12779/) (2025)
- [Visual-tactile Fusion for Transparent Object Grasping in Complex Backgrounds](https://mlsanity.com/paper/2211.16693/) (2022)
- [Active 6D Pose Estimation for Textureless Objects using Multi-View RGB Frames](https://mlsanity.com/paper/2503.03726/) (2025)
- [TransNormal: Dense Visual Semantics for Diffusion-based Transparent Object Normal Estimation](https://mlsanity.com/paper/2602.00839/) (2026)
- [TransNet: Transparent Object Manipulation Through Category-Level Pose Estimation](https://mlsanity.com/paper/2307.12400/) (2023)
- [Transparent Object Depth Completion](https://mlsanity.com/paper/2405.15299/) (2024)
- [Spiers Memorial Lecture: How to do impactful research in artificial intelligence for chemistry and materials science](https://mlsanity.com/paper/2409.10304/) (2024)
- [Object Pose Estimation Using Implicit Representation For Transparent Objects](https://mlsanity.com/paper/2410.13465/) (2024)
- [UASTrack: A Unified Adaptive Selection Framework with Modality-Customization in Single Object Tracking](https://mlsanity.com/paper/2502.18220/) (2025)
- [Shaken, Not Stirred: A Novel Dataset for Visual Understanding of Glasses in Human-Robot Bartending Tasks](https://mlsanity.com/paper/2503.04308/) (2025)
- [StarryGazer: Leveraging Monocular Depth Estimation Models for Domain-Agnostic Single Depth Image Completion](https://mlsanity.com/paper/2512.13147/) (2025)
- [AnchorD: Metric Grounding of Monocular Depth Using Factor Graphs](https://mlsanity.com/paper/2605.02667/) (2026)
- [TransBiolab: A Real-World Multi-View Dataset of Cluttered Transparent Biomedical Objects](https://mlsanity.com/paper/2607.21071/) (2026)
- [Tabletop Transparent Scene Reconstruction via Epipolar-Guided Optical Flow with Monocular Depth Completion Prior](https://mlsanity.com/paper/2310.09956/) (2023)
- [Semantic Segmentation of Transparent and Opaque Drinking Glasses with the Help of Zero-shot Learning](https://mlsanity.com/paper/2503.15004/) (2025)
- [GAA-TSO: Geometry-Aware Assisted Depth Completion for Transparent and Specular Objects](https://mlsanity.com/paper/2503.17106/) (2025)

## Abstract (arXiv)

The basis of many object manipulation algorithms is RGB-D input. Yet, commodity RGB-D sensors can only provide distorted depth maps for a wide range of transparent objects due light refraction and absorption. To tackle the perception challenges posed by transparent objects, we propose TranspareNet, a joint point cloud and depth completion method, with the ability to complete the depth of transparent objects in cluttered and complex scenes, even with partially filled fluid contents within the vessels. To address the shortcomings of existing transparent object data collection schemes in literature, we also propose an automated dataset creation workflow that consists of robot-controlled image collection and vision-based automatic annotation. Through this automated workflow, we created Toronto Transparent Objects Depth Dataset (TODD), which consists of nearly 15000 RGB-D images. Our experimental evaluation demonstrates that TranspareNet outperforms existing state-of-the-art depth completion methods on multiple datasets, including ClearGrasp, and that it also handles cluttered scenes when trained on TODD. Code and dataset will be released at https://www.pair.toronto.edu/TranspareNet/