# TransTouch: Learning Transparent Objects Depth Sensing Through Sparse Touches

arXiv: https://arxiv.org/abs/2309.09427 · HTML: https://mlsanity.com/paper/2309.09427/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Builds on (most cited first, 19 of 19)

- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness](https://mlsanity.com/paper/1911.12012/) (2019)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Learning to Compare Image Patches via Convolutional Neural Networks](https://mlsanity.com/paper/1504.03641/) (2015)
- [Computing the Stereo Matching Cost with a Convolutional Neural Network](https://mlsanity.com/paper/1409.4326/) (2014)
- [RGB-D Local Implicit Function for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2104.00622/) (2021)
- [TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline](https://mlsanity.com/paper/2202.08471/) (2022)
- [Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects](https://mlsanity.com/paper/2208.03792/) (2022)
- [3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2007.03778/) (2020)
- [Active 3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2107.09584/) (2021)
- [3D Shape Perception from Monocular Vision, Touch, and Shape Priors](https://mlsanity.com/paper/1808.03247/) (2018)
- [Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation](https://mlsanity.com/paper/2201.11924/) (2022)
- [Multi-Modal Geometric Learning for Grasping and Manipulation](https://mlsanity.com/paper/1803.07671/) (2018)
- [Active Visuo-Haptic Object Shape Completion](https://mlsanity.com/paper/2203.09149/) (2022)
- [ActiveZero: Mixed Domain Learning for Active Stereovision with Zero Annotation](https://mlsanity.com/paper/2112.02772/) (2021)
- [Where Shall I Touch? Vision-Guided Tactile Poking for Transparent Object Grasping](https://mlsanity.com/paper/2208.09743/) (2022)

## Built on by (most cited first, 2 of 2)

- [Object Pose Estimation through Dexterous Touch](https://mlsanity.com/paper/2509.13591/) (2025)
- [Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly](https://mlsanity.com/paper/2603.20290/) (2026)

## Abstract (arXiv)

Transparent objects are common in daily life. However, depth sensing for transparent objects remains a challenging problem. While learning-based methods can leverage shape priors to improve the sensing quality, the labor-intensive data collection in the real world and the sim-to-real domain gap restrict these methods' scalability. In this paper, we propose a method to finetune a stereo network with sparse depth labels automatically collected using a probing system with tactile feedback. We present a novel utility function to evaluate the benefit of touches. By approximating and optimizing the utility function, we can optimize the probing locations given a fixed touching budget to better improve the network's performance on real objects. We further combine tactile depth supervision with a confidence-based regularization to prevent over-fitting during finetuning. To evaluate the effectiveness of our method, we construct a real-world dataset including both diffuse and transparent objects. Experimental results on this dataset show that our method can significantly improve real-world depth sensing accuracy, especially for transparent objects.