# 3D Shape Perception from Monocular Vision, Touch, and Shape Priors

arXiv: https://arxiv.org/abs/1808.03247 · HTML: https://mlsanity.com/paper/1808.03247/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2018.

## Built on by (most cited first, 30 of 31)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play](https://mlsanity.com/paper/2303.12076/) (2023)
- [Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing](https://mlsanity.com/paper/2312.01853/) (2023)
- [See to Touch: Learning Tactile Dexterity through Visual Incentives](https://mlsanity.com/paper/2309.12300/) (2023)
- [Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2408.04380/) (2024)
- [Active 3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2107.09584/) (2021)
- [Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation](https://mlsanity.com/paper/2312.13469/) (2023)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [Tac2Pose: Tactile Object Pose Estimation from the First Touch](https://mlsanity.com/paper/2204.11701/) (2022)
- [TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing](https://mlsanity.com/paper/2311.12602/) (2023)
- [ViHOPE: Visuotactile In-Hand Object 6D Pose Estimation with Shape Completion](https://mlsanity.com/paper/2309.05662/) (2023)
- [Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing](https://mlsanity.com/paper/2210.09297/) (2022)
- [Visual-Tactile Sensing for In-Hand Object Reconstruction](https://mlsanity.com/paper/2303.14498/) (2023)
- [Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfaces](https://mlsanity.com/paper/2403.20275/) (2024)
- [Active Visuo-Haptic Object Shape Completion](https://mlsanity.com/paper/2203.09149/) (2022)
- [Dynamic Reconstruction of Hand-Object Interaction with Distributed Force-aware Contact Representation](https://mlsanity.com/paper/2411.09572/) (2024)
- [Touch-based Curiosity for Sparse-Reward Tasks](https://mlsanity.com/paper/2104.00442/) (2021)
- [Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation](https://mlsanity.com/paper/2412.06785/) (2024)
- [GelSLAM: A Real-time, High-Fidelity, and Robust 3D Tactile SLAM System](https://mlsanity.com/paper/2508.15990/) (2025)
- [Elastic Tactile Simulation Towards Tactile-Visual Perception](https://mlsanity.com/paper/2108.05013/) (2021)
- [Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing](https://mlsanity.com/paper/2210.04026/) (2022)
- [Learning Intuitive Physics with Multimodal Generative Models](https://mlsanity.com/paper/2101.04454/) (2021)
- [Multimodal perception for dexterous manipulation](https://mlsanity.com/paper/2112.14298/) (2021)
- [TransTouch: Learning Transparent Objects Depth Sensing Through Sparse Touches](https://mlsanity.com/paper/2309.09427/) (2023)
- [FusionSense: Bridging Common Sense, Vision, and Touch for Robust Sparse-View Reconstruction](https://mlsanity.com/paper/2410.08282/) (2024)
- [A Bayesian Framework for Active Tactile Object Recognition, Pose Estimation and Shape Transfer Learning](https://mlsanity.com/paper/2409.06912/) (2024)
- [VisTa3D: A Dataset and Benchmark for Thin Object Reconstruction from Vision, Tactile, and 3D Point Clouds](https://mlsanity.com/paper/2608.20740/) (2026)
- [Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition](https://mlsanity.com/paper/2311.11287/) (2023)
- [UniTac2Pose: A Unified Approach Learned in Simulation for Category-level Visuotactile In-hand Pose Estimation](https://mlsanity.com/paper/2509.15934/) (2025)
- [Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch](https://mlsanity.com/paper/2604.09100/) (2026)

## Abstract (arXiv)

Perceiving accurate 3D object shape is important for robots to interact with the physical world. Current research along this direction has been primarily relying on visual observations. Vision, however useful, has inherent limitations due to occlusions and the 2D-3D ambiguities, especially for perception with a monocular camera. In contrast, touch gets precise local shape information, though its efficiency for reconstructing the entire shape could be low. In this paper, we propose a novel paradigm that efficiently perceives accurate 3D object shape by incorporating visual and tactile observations, as well as prior knowledge of common object shapes learned from large-scale shape repositories. We use vision first, applying neural networks with learned shape priors to predict an object's 3D shape from a single-view color image. We then use tactile sensing to refine the shape; the robot actively touches the object regions where the visual prediction has high uncertainty. Our method efficiently builds the 3D shape of common objects from a color image and a small number of tactile explorations (around 10). Our setup is easy to apply and has potentials to help robots better perform grasping or manipulation tasks on real-world objects.