# Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features

arXiv: https://arxiv.org/abs/2209.13042 · HTML: https://mlsanity.com/paper/2209.13042/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2022.

## Built on by (most cited first, 30 of 38)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [A Touch, Vision, and Language Dataset for Multimodal Alignment](https://mlsanity.com/paper/2402.13232/) (2024)
- [AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors](https://mlsanity.com/paper/2502.12191/) (2025)
- [Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training](https://mlsanity.com/paper/2401.12024/) (2024)
- [ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data](https://mlsanity.com/paper/2406.19464/) (2024)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities](https://mlsanity.com/paper/2412.11694/) (2024)
- [OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All](https://mlsanity.com/paper/2405.16108/) (2024)
- [Continual Multimodal Contrastive Learning](https://mlsanity.com/paper/2503.14963/) (2025)
- [VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policies](https://mlsanity.com/paper/2403.11898/) (2024)
- [AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception](https://mlsanity.com/paper/2602.09617/) (2026)
- [TextToucher: Fine-Grained Text-to-Touch Generation](https://mlsanity.com/paper/2409.05427/) (2024)
- [Neural Brain: A Neuroscience-inspired Framework for Embodied Agents](https://mlsanity.com/paper/2505.07634/) (2025)
- [M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation](https://mlsanity.com/paper/2401.17032/) (2024)
- [Sensor-Invariant Tactile Representation](https://mlsanity.com/paper/2502.19638/) (2025)
- [Universal Visuo-Tactile Video Understanding for Embodied Interaction](https://mlsanity.com/paper/2505.22566/) (2025)
- [ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representations](https://mlsanity.com/paper/2506.20757/) (2025)
- [Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Dataset](https://mlsanity.com/paper/2403.09813/) (2024)
- [UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation](https://mlsanity.com/paper/2606.31451/) (2026)
- [X-Capture: An Open-Source Portable Device for Multi-Sensory Learning](https://mlsanity.com/paper/2504.02318/) (2025)
- [DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter](https://mlsanity.com/paper/2602.05513/) (2026)
- [Tactile MNIST: Benchmarking Active Tactile Perception](https://mlsanity.com/paper/2506.06361/) (2025)
- [RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data](https://mlsanity.com/paper/2505.14270/) (2025)
- [ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers](https://mlsanity.com/paper/2505.20032/) (2025)
- [Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities](https://mlsanity.com/paper/2511.11512/) (2025)
- [TaCo: A Benchmark for Lossless and Lossy Codecs of Heterogeneous Tactile Data](https://mlsanity.com/paper/2602.09893/) (2026)
- [Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions](https://mlsanity.com/paper/2604.11579/) (2026)
- [τ: Learning Touch-Augmented Vision-Language-Action Models from Future Visual Supervision](https://mlsanity.com/paper/2607.24485/) (2026)
- [TacForcing: Streaming Action Generation with Execution-Time Tactile Feedback](https://mlsanity.com/paper/2608.25798/) (2026)

## Abstract (arXiv)

Humans make extensive use of vision and touch as complementary senses, with vision providing global information about the scene and touch measuring local information during manipulation without suffering from occlusions. While prior work demonstrates the efficacy of tactile sensing for precise manipulation of deformables, they typically rely on supervised, human-labeled datasets. We propose Self-Supervised Visuo-Tactile Pretraining (SSVTP), a framework for learning multi-task visuo-tactile representations in a self-supervised manner through cross-modal supervision. We design a mechanism that enables a robot to autonomously collect precisely spatially-aligned visual and tactile image pairs, then train visual and tactile encoders to embed these pairs into a shared latent space using cross-modal contrastive loss. We apply this latent space to downstream perception and control of deformable garments on flat surfaces, and evaluate the flexibility of the learned representations without fine-tuning on 5 tasks: feature classification, contact localization, anomaly detection, feature search from a visual query (e.g., garment feature localization under occlusion), and edge following along cloth edges. The pretrained representations achieve a 73-100% success rate on these 5 tasks.