# Connecting Look and Feel: Associating the visual and tactile properties of physical materials

arXiv: https://arxiv.org/abs/1704.03822 · HTML: https://mlsanity.com/paper/1704.03822/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2017.

## Built on by (most cited first, 22 of 22)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback](https://mlsanity.com/paper/2303.07997/) (2023)
- [Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning](https://mlsanity.com/paper/2204.00117/) (2022)
- [Dynamic Modeling of Hand-Object Interactions via Tactile Sensing](https://mlsanity.com/paper/2109.04378/) (2021)
- [Controllable Visual-Tactile Synthesis](https://mlsanity.com/paper/2305.03051/) (2023)
- [Sensor-Invariant Tactile Representation](https://mlsanity.com/paper/2502.19638/) (2025)
- [Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Losses](https://mlsanity.com/paper/2107.05468/) (2021)
- [Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Dataset](https://mlsanity.com/paper/2403.09813/) (2024)
- [Learning Intuitive Physics with Multimodal Generative Models](https://mlsanity.com/paper/2101.04454/) (2021)
- [Tactile MNIST: Benchmarking Active Tactile Perception](https://mlsanity.com/paper/2506.06361/) (2025)
- [New York Smells: A Large Multimodal Dataset for Olfaction](https://mlsanity.com/paper/2511.20544/) (2025)
- [ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers](https://mlsanity.com/paper/2505.20032/) (2025)
- [TaCo: A Benchmark for Lossless and Lossy Codecs of Heterogeneous Tactile Data](https://mlsanity.com/paper/2602.09893/) (2026)
- [Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions](https://mlsanity.com/paper/2604.11579/) (2026)
- [Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms](https://mlsanity.com/paper/2605.17336/) (2026)
- [Using deep learning to enhance electronic service quality: Application to real estate websites](https://mlsanity.com/paper/2408.06364/) (2024)
- [MLLM-Fabric: Multimodal Large Language Model-Driven Robotic Framework for Fabric Sorting and Selection](https://mlsanity.com/paper/2507.04351/) (2025)
- [RCT: A Robot-Collected Touch-Vision-Language Dataset for Tactile Generalization](https://mlsanity.com/paper/2606.31694/) (2026)
- [Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs](https://mlsanity.com/paper/2607.00302/) (2026)
- [What Images Cannot Say: Language-Guided Olfactory Representation Learning](https://mlsanity.com/paper/2607.06402/) (2026)

## Abstract (arXiv)

For machines to interact with the physical world, they must understand the physical properties of objects and materials they encounter. We use fabrics as an example of a deformable material with a rich set of mechanical properties. A thin flexible fabric, when draped, tends to look different from a heavy stiff fabric. It also feels different when touched. Using a collection of 118 fabric sample, we captured color and depth images of draped fabrics along with tactile data from a high resolution touch sensor. We then sought to associate the information from vision and touch by jointly training CNNs across the three modalities. Through the CNN, each input, regardless of the modality, generates an embedding vector that records the fabric's physical property. By comparing the embeddings, our system is able to look at a fabric image and predict how it will feel, and vice versa. We also show that a system jointly trained on vision and touch data can outperform a similar system trained only on visual data when tested purely with visual inputs.