# Active Visuo-Haptic Object Shape Completion

arXiv: https://arxiv.org/abs/2203.09149 · HTML: https://mlsanity.com/paper/2203.09149/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis](https://mlsanity.com/paper/1612.00101/) (2016)
- [Revisiting Active Perception](https://mlsanity.com/paper/1603.02729/) (2016)
- [High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference](https://mlsanity.com/paper/1709.07599/) (2017)
- [Shape Completion Enabled Robotic Grasping](https://mlsanity.com/paper/1609.08546/) (2016)
- [3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2007.03778/) (2020)
- [Active 3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2107.09584/) (2021)
- [3D Shape Perception from Monocular Vision, Touch, and Shape Priors](https://mlsanity.com/paper/1808.03247/) (2018)
- [Robotic Tactile Perception of Object Properties: A Review](https://mlsanity.com/paper/1711.03810/) (2017)
- [Robust Grasp Planning Over Uncertain Shape Completions](https://mlsanity.com/paper/1903.00645/) (2019)
- [Multi-Modal Geometric Learning for Grasping and Manipulation](https://mlsanity.com/paper/1803.07671/) (2018)

## Built on by (most cited first, 11 of 11)

- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing](https://mlsanity.com/paper/2311.12602/) (2023)
- [Hierarchical Graph Neural Networks for Proprioceptive 6D Pose Estimation of In-hand Objects](https://mlsanity.com/paper/2306.15858/) (2023)
- [GelSLAM: A Real-time, High-Fidelity, and Robust 3D Tactile SLAM System](https://mlsanity.com/paper/2508.15990/) (2025)
- [Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing](https://mlsanity.com/paper/2210.04026/) (2022)
- [Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction](https://mlsanity.com/paper/2505.13091/) (2025)
- [TransTouch: Learning Transparent Objects Depth Sensing Through Sparse Touches](https://mlsanity.com/paper/2309.09427/) (2023)
- [Imagine2touch: Predictive Tactile Sensing for Robotic Manipulation using Efficient Low-Dimensional Signals](https://mlsanity.com/paper/2405.01192/) (2024)
- [PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation](https://mlsanity.com/paper/2403.15107/) (2024)
- [RUMI: Rummaging Using Mutual Information](https://mlsanity.com/paper/2408.10450/) (2024)
- [Object Pose Estimation through Dexterous Touch](https://mlsanity.com/paper/2509.13591/) (2025)

## Abstract (arXiv)

Recent advancements in object shape completion have enabled impressive object reconstructions using only visual input. However, due to self-occlusion, the reconstructions have high uncertainty in the occluded object parts, which negatively impacts the performance of downstream robotic tasks such as grasping. In this work, we propose an active visuo-haptic shape completion method called Act-VH that actively computes where to touch the objects based on the reconstruction uncertainty. Act-VH reconstructs objects from point clouds and calculates the reconstruction uncertainty using IGR, a recent state-of-the-art implicit surface deep neural network. We experimentally evaluate the reconstruction accuracy of Act-VH against five baselines in simulation and in the real world. We also propose a new simulation environment for this purpose. The results show that Act-VH outperforms all baselines and that an uncertainty-driven haptic exploration policy leads to higher reconstruction accuracy than a random policy and a policy driven by Gaussian Process Implicit Surfaces. As a final experiment, we evaluate Act-VH and the best reconstruction baseline on grasping 10 novel objects. The results show that Act-VH reaches a significantly higher grasp success rate than the baseline on all objects. Together, this work opens up the door for using active visuo-haptic shape completion in more complex cluttered scenes.