# Planning Visual-Tactile Precision Grasps via Complementary Use of Vision and Touch

arXiv: https://arxiv.org/abs/2212.08604 · HTML: https://mlsanity.com/paper/2212.08604/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 16 Dec 2022.

## Builds on (most cited first, 22 of 22)

- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations](https://mlsanity.com/paper/2104.01542/) (2021)
- [Data-Driven Grasp Synthesis - A Survey](https://mlsanity.com/paper/1309.2660/) (2013)
- [More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch](https://mlsanity.com/paper/1805.11085/) (2018)
- [The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?](https://mlsanity.com/paper/1710.05512/) (2017)
- [Learning Continuous 3D Reconstructions for Geometrically Aware Grasping](https://mlsanity.com/paper/1910.00983/) (2019)
- [DDGC: Generative Deep Dexterous Grasping in Clutter](https://mlsanity.com/paper/2103.04783/) (2021)
- [Multi-FinGAN: Generative Coarse-To-Fine Sampling of Multi-Finger Grasps](https://mlsanity.com/paper/2012.09696/) (2020)
- [Learning to Grasp Without Seeing](https://mlsanity.com/paper/1805.04201/) (2018)
- [Tactile Regrasp: Grasp Adjustments via Simulated Tactile Transformations](https://mlsanity.com/paper/1803.01940/) (2018)
- [Leveraging Contact Forces for Learning to Grasp](https://mlsanity.com/paper/1809.07004/) (2018)
- [Multi-Fingered Active Grasp Learning](https://mlsanity.com/paper/2006.05264/) (2020)
- [MAT: Multi-Fingered Adaptive Tactile Grasping via Deep Reinforcement Learning](https://mlsanity.com/paper/1909.04787/) (2019)
- [Simultaneous Tactile Exploration and Grasp Refinement for Unknown Objects](https://mlsanity.com/paper/2103.00655/) (2021)
- [Robust Learning of Tactile Force Estimation through Robot Interaction](https://mlsanity.com/paper/1810.06187/) (2018)
- [Benchmarking In-Hand Manipulation](https://mlsanity.com/paper/2001.03070/) (2020)
- [Modeling Grasp Type Improves Learning-Based Grasp Planning](https://mlsanity.com/paper/1901.02992/) (2019)
- [Learning better generative models for dexterous, single-view grasping of novel objects](https://mlsanity.com/paper/1907.06053/) (2019)
- [Modeling Grasp Motor Imagery through Deep Conditional Generative Models](https://mlsanity.com/paper/1701.03041/) (2017)
- [Pixel-Attentive Policy Gradient for Multi-Fingered Grasping in Cluttered Scenes](https://mlsanity.com/paper/1903.03227/) (2019)
- [In-Hand Object Stabilization by Independent Finger Control](https://mlsanity.com/paper/1806.05031/) (2018)

## Built on by (most cited first, 2 of 2)

- [Robust Bayesian Scene Reconstruction with Retrieval-Augmented Priors for Precise Grasping and Planning](https://mlsanity.com/paper/2411.19461/) (2024)
- [D3Grasp: Diverse and Deformable Dexterous Grasping for General Objects](https://mlsanity.com/paper/2509.19892/) (2025)

## Abstract (arXiv)

Reliably planning fingertip grasps for multi-fingered hands lies as a key challenge for many tasks including tool use, insertion, and dexterous in-hand manipulation. This task becomes even more difficult when the robot lacks an accurate model of the object to be grasped. Tactile sensing offers a promising approach to account for uncertainties in object shape. However, current robotic hands tend to lack full tactile coverage. As such, a problem arises of how to plan and execute grasps for multi-fingered hands such that contact is made with the area covered by the tactile sensors. To address this issue, we propose an approach to grasp planning that explicitly reasons about where the fingertips should contact the estimated object surface while maximizing the probability of grasp success. Key to our method's success is the use of visual surface estimation for initial planning to encode the contact constraint. The robot then executes this plan using a tactile-feedback controller that enables the robot to adapt to online estimates of the object's surface to correct for errors in the initial plan. Importantly, the robot never explicitly integrates object pose or surface estimates between visual and tactile sensing, instead it uses the two modalities in complementary ways. Vision guides the robots motion prior to contact; touch updates the plan when contact occurs differently than predicted from vision. We show that our method successfully synthesises and executes precision grasps for previously unseen objects using surface estimates from a single camera view. Further, our approach outperforms a state of the art multi-fingered grasp planner, while also beating several baselines we propose.