# MVTrans: Multi-View Perception of Transparent Objects

arXiv: https://arxiv.org/abs/2302.11683 · HTML: https://mlsanity.com/paper/2302.11683/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 22 Feb 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [ACRONYM: A Large-Scale Grasp Dataset Based on Simulation](https://mlsanity.com/paper/2011.09584/) (2020)
- [Deep Depth Completion of a Single RGB-D Image](https://mlsanity.com/paper/1803.09326/) (2018)
- [Segmenting Transparent Objects in the Wild](https://mlsanity.com/paper/2003.13948/) (2020)
- [Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation](https://mlsanity.com/paper/1804.06534/) (2018)
- [CoPhy: Counterfactual Learning of Physical Dynamics](https://mlsanity.com/paper/1909.12000/) (2019)
- [KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects](https://mlsanity.com/paper/1912.02805/) (2019)
- [RGB-D Local Implicit Function for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2104.00622/) (2021)
- [Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks](https://mlsanity.com/paper/2011.13118/) (2020)
- [Enhanced Boundary Learning for Glass-like Object Segmentation](https://mlsanity.com/paper/2103.15734/) (2021)
- [MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion](https://mlsanity.com/paper/2004.04336/) (2020)
- [ClearPose: Large-scale Transparent Object Dataset and Benchmark](https://mlsanity.com/paper/2203.03890/) (2022)
- [The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation](https://mlsanity.com/paper/1907.13236/) (2019)
- [Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects](https://mlsanity.com/paper/2110.00087/) (2021)
- [Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World](https://mlsanity.com/paper/2107.03172/) (2021)
- [SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo](https://mlsanity.com/paper/2106.16118/) (2021)
- [RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis](https://mlsanity.com/paper/1909.12573/) (2019)
- [SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments](https://mlsanity.com/paper/2108.06180/) (2021)
- [FakeMix Augmentation Improves Transparent Object Detection](https://mlsanity.com/paper/2103.13279/) (2021)
- [AVoE: A Synthetic 3D Dataset on Understanding Violation of Expectation for Artificial Cognition](https://mlsanity.com/paper/2110.05836/) (2021)
- [Predicting 3D shapes, masks, and properties of materials, liquids, and objects inside transparent containers, using the TransProteus CGI dataset](https://mlsanity.com/paper/2109.07577/) (2021)

## Built on by (most cited first, 14 of 14)

- [Manipulate-Anything: Automating Real-World Robots using Vision-Language Models](https://mlsanity.com/paper/2406.18915/) (2024)
- [ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization](https://mlsanity.com/paper/2401.06949/) (2024)
- [ClearDepth: Enhanced Stereo Perception of Transparent Objects for Robotic Manipulation](https://mlsanity.com/paper/2409.08926/) (2024)
- [RoboChemist: Long-Horizon and Safety-Compliant Robotic Chemical Experimentation](https://mlsanity.com/paper/2509.08820/) (2025)
- [Monocular Depth Estimation and Segmentation for Transparent Object with Iterative Semantic and Geometric Fusion](https://mlsanity.com/paper/2502.14616/) (2025)
- [MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation](https://mlsanity.com/paper/2601.06874/) (2026)
- [SAID-NeRF: Segmentation-AIDed NeRF for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2403.19607/) (2024)
- [Residual-NeRF: Learning Residual NeRFs for Transparent Object Manipulation](https://mlsanity.com/paper/2405.06181/) (2024)
- [Transparent Object Depth Completion](https://mlsanity.com/paper/2405.15299/) (2024)
- [Shaken, Not Stirred: A Novel Dataset for Visual Understanding of Glasses in Human-Robot Bartending Tasks](https://mlsanity.com/paper/2503.04308/) (2025)
- [EGSA-PT:Edge-Guided Spatial Attention with Progressive Training for Monocular Depth Estimation and Segmentation of Transparent Objects](https://mlsanity.com/paper/2511.14970/) (2025)
- [ReFlow6D: Refraction-Guided Transparent Object 6D Pose Estimation via Intermediate Representation Learning](https://mlsanity.com/paper/2412.20830/) (2024)
- [Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly](https://mlsanity.com/paper/2603.20290/) (2026)
- [AISPO: Enhancing Depth Reliability for Robotic Manipulation of Non-Lambertian Objects via Affine-Invariant Shape Prior](https://mlsanity.com/paper/2606.25503/) (2026)

## Abstract (arXiv)

Transparent object perception is a crucial skill for applications such as robot manipulation in household and laboratory settings. Existing methods utilize RGB-D or stereo inputs to handle a subset of perception tasks including depth and pose estimation. However, transparent object perception remains to be an open problem. In this paper, we forgo the unreliable depth map from RGB-D sensors and extend the stereo based method. Our proposed method, MVTrans, is an end-to-end multi-view architecture with multiple perception capabilities, including depth estimation, segmentation, and pose estimation. Additionally, we establish a novel procedural photo-realistic dataset generation pipeline and create a large-scale transparent object detection dataset, Syn-TODD, which is suitable for training networks with all three modalities, RGB-D, stereo and multi-view RGB. Project Site: https://ac-rad.github.io/MVTrans/