# Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching

arXiv: https://arxiv.org/abs/1903.03591 · HTML: https://mlsanity.com/paper/1903.03591/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2019.

## Built on by (most cited first, 16 of 16)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [Generating Visual Scenes from Touch](https://mlsanity.com/paper/2309.15117/) (2023)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [AcTExplore: Active Tactile Exploration of Unknown Objects](https://mlsanity.com/paper/2310.08745/) (2023)
- [Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skins](https://mlsanity.com/paper/2410.17246/) (2024)
- [Controllable Visual-Tactile Synthesis](https://mlsanity.com/paper/2305.03051/) (2023)
- [M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation](https://mlsanity.com/paper/2401.17032/) (2024)
- [Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Dataset](https://mlsanity.com/paper/2403.09813/) (2024)
- [TactoFind: A Tactile Only System for Object Retrieval](https://mlsanity.com/paper/2303.13482/) (2023)
- [eFlesh: Highly customizable Magnetic Touch Sensing using Cut-Cell Microstructures](https://mlsanity.com/paper/2506.09994/) (2025)
- [Multimodal perception for dexterous manipulation](https://mlsanity.com/paper/2112.14298/) (2021)
- [Latent Object Characteristics Recognition with Visual to Haptic-Audio Cross-modal Transfer Learning](https://mlsanity.com/paper/2403.10689/) (2024)
- [Test-Time Adaptation for Tactile-Vision-Language Models](https://mlsanity.com/paper/2602.15873/) (2026)
- [MultiDiffSense: Diffusion-Based Multi-Modal Visuo-Tactile Image Generation Conditioned on Object Shape and Contact Pose](https://mlsanity.com/paper/2602.19348/) (2026)

## Abstract (arXiv)

Much of the literature on robotic perception focuses on the visual modality. Vision provides a global observation of a scene, making it broadly useful. However, in the domain of robotic manipulation, vision alone can sometimes prove inadequate: in the presence of occlusions or poor lighting, visual object identification might be difficult. The sense of touch can provide robots with an alternative mechanism for recognizing objects. In this paper, we study the problem of touch-based instance recognition. We propose a novel framing of the problem as multi-modal recognition: the goal of our system is to recognize, given a visual and tactile observation, whether or not these observations correspond to the same object. To our knowledge, our work is the first to address this type of multi-modal instance recognition problem on such a large-scale with our analysis spanning 98 different objects. We employ a robot equipped with two GelSight touch sensors, one on each finger, and a self-supervised, autonomous data collection procedure to collect a dataset of tactile observations and images. Our experimental results show that it is possible to accurately recognize object instances by touch alone, including instances of novel objects that were never seen during training. Our learned model outperforms other methods on this complex task, including that of human volunteers.