# More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch

arXiv: https://arxiv.org/abs/1805.11085 · HTML: https://mlsanity.com/paper/1805.11085/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2018.

## Built on by (most cited first, 30 of 102)

- [General In-Hand Object Rotation with Vision and Touch](https://mlsanity.com/paper/2309.09979/) (2023)
- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation](https://mlsanity.com/paper/2211.09423/) (2022)
- [Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers](https://mlsanity.com/paper/2107.03996/) (2021)
- [Learning Visuotactile Skills with Two Multifingered Hands](https://mlsanity.com/paper/2404.16823/) (2024)
- [ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer](https://mlsanity.com/paper/2204.02389/) (2022)
- [3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing](https://mlsanity.com/paper/2410.24091/) (2024)
- [A Touch, Vision, and Language Dataset for Multimodal Alignment](https://mlsanity.com/paper/2402.13232/) (2024)
- [ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations](https://mlsanity.com/paper/2109.07991/) (2021)
- [See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2212.03858/) (2022)
- [Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play](https://mlsanity.com/paper/2303.12076/) (2023)
- [Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing](https://mlsanity.com/paper/2312.01853/) (2023)
- [TLA: Tactile-Language-Action Model for Contact-Rich Manipulation](https://mlsanity.com/paper/2503.08548/) (2025)
- [Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks](https://mlsanity.com/paper/2406.13640/) (2024)
- [Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding](https://mlsanity.com/paper/2501.04693/) (2025)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World](https://mlsanity.com/paper/2401.08577/) (2024)
- [OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensing](https://mlsanity.com/paper/2508.08706/) (2025)
- [ReSkin: versatile, replaceable, lasting tactile skins](https://mlsanity.com/paper/2111.00071/) (2021)
- [See to Touch: Learning Tactile Dexterity through Visual Incentives](https://mlsanity.com/paper/2309.12300/) (2023)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger](https://mlsanity.com/paper/2106.08851/) (2021)
- [MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation](https://mlsanity.com/paper/2310.16917/) (2023)
- [GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training](https://mlsanity.com/paper/2507.13097/) (2025)
- [Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training](https://mlsanity.com/paper/2401.12024/) (2024)
- [Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper](https://mlsanity.com/paper/2507.15062/) (2025)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation](https://mlsanity.com/paper/2101.11812/) (2021)
- [AnySkin: Plug-and-play Skin Sensing for Robotic Touch](https://mlsanity.com/paper/2409.08276/) (2024)

## Abstract (arXiv)

For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this paper, we investigate how a robot can learn to use tactile information to iteratively and efficiently adjust its grasp. To this end, we propose an end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data. This model -- a deep, multimodal convolutional network -- predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions. Our approach requires neither calibration of the tactile sensors, nor any analytical modeling of contact forces, thus reducing the engineering effort required to obtain efficient grasping policies. We train our model with data from about 6,450 grasping trials on a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger. Across extensive experiments, our approach outperforms a variety of baselines at (i) estimating grasp adjustment outcomes, (ii) selecting efficient grasp adjustments for quick grasping, and (iii) reducing the amount of force applied at the fingers, while maintaining competitive performance. Finally, we study the choices made by our model and show that it has successfully acquired useful and interpretable grasping behaviors.