# Tabletop Transparent Scene Reconstruction via Epipolar-Guided Optical Flow with Monocular Depth Completion Prior

arXiv: https://arxiv.org/abs/2310.09956 · HTML: https://mlsanity.com/paper/2310.09956/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 15 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects](https://mlsanity.com/paper/2110.14217/) (2021)
- [Segmenting Transparent Objects in the Wild](https://mlsanity.com/paper/2003.13948/) (2020)
- [KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects](https://mlsanity.com/paper/1912.02805/) (2019)
- [RGB-D Local Implicit Function for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2104.00622/) (2021)
- [Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects](https://mlsanity.com/paper/2208.03792/) (2022)
- [ClearPose: Large-scale Transparent Object Dataset and Benchmark](https://mlsanity.com/paper/2203.03890/) (2022)
- [StereOBJ-1M: Large-scale Stereo Image Dataset for 6D Object Pose Estimation](https://mlsanity.com/paper/2109.10115/) (2021)
- [Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects](https://mlsanity.com/paper/2110.00087/) (2021)
- [Through the Looking Glass: Neural 3D Reconstruction of Transparent Shapes](https://mlsanity.com/paper/2004.10904/) (2020)
- [Full 3D Reconstruction of Transparent Objects](https://mlsanity.com/paper/1805.03482/) (2018)
- [Differentiable Refraction-Tracing for Mesh Reconstruction of Transparent Objects](https://mlsanity.com/paper/2009.09144/) (2020)
- [Hybrid Mesh-neural Representation for 3D Transparent Object Reconstruction](https://mlsanity.com/paper/2203.12613/) (2022)
- [LIT: Light-field Inference of Transparency for Refractive Object Localization](https://mlsanity.com/paper/1910.00721/) (2019)
- [GlassLoc: Plenoptic Grasp Pose Detection in Transparent Clutter](https://mlsanity.com/paper/1909.04269/) (2019)
- [Online Object Model Reconstruction and Reuse for Lifelong Improvement of Robot Manipulation](https://mlsanity.com/paper/2109.13910/) (2021)
- [Plenoptic Monte Carlo Object Localization for Robot Grasping under Layered Translucency](https://mlsanity.com/paper/1806.09769/) (2018)

## Abstract (arXiv)

Reconstructing transparent objects using affordable RGB-D cameras is a persistent challenge in robotic perception due to inconsistent appearances across views in the RGB domain and inaccurate depth readings in each single-view. We introduce a two-stage pipeline for reconstructing transparent objects tailored for mobile platforms. In the first stage, off-the-shelf monocular object segmentation and depth completion networks are leveraged to predict the depth of transparent objects, furnishing single-view shape prior. Subsequently, we propose Epipolar-guided Optical Flow (EOF) to fuse several single-view shape priors from the first stage to a cross-view consistent 3D reconstruction given camera poses estimated from opaque part of the scene. Our key innovation lies in EOF which employs boundary-sensitive sampling and epipolar-line constraints into optical flow to accurately establish 2D correspondences across multiple views on transparent objects. Quantitative evaluations demonstrate that our pipeline significantly outperforms baseline methods in 3D reconstruction quality, paving the way for more adept robotic perception and interaction with transparent objects.