# Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation

arXiv: https://arxiv.org/abs/2506.14754 · HTML: https://mlsanity.com/paper/2506.14754/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2025.

## Built on by (most cited first, 11 of 11)

- [Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations](https://mlsanity.com/paper/2511.16661/) (2025)
- [AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception](https://mlsanity.com/paper/2602.09617/) (2026)
- [Learning Dexterous Manipulation Skills from Imperfect Simulations](https://mlsanity.com/paper/2512.02011/) (2025)
- [DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter](https://mlsanity.com/paper/2602.05513/) (2026)
- [Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots](https://mlsanity.com/paper/2607.13522/) (2026)
- [Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation](https://mlsanity.com/paper/2605.28812/) (2026)
- [Objects as Audio-Visual Modal Sound Fields](https://mlsanity.com/paper/2608.05145/) (2026)
- [The Potential of Haptic Foundation Models](https://mlsanity.com/paper/2608.28664/) (2026)
- [BIDETA: Brain-Inspired Data-Efficient Tactile Adaptation for Unseen Sensors](https://mlsanity.com/paper/2609.08673/) (2026)
- [Tactile-JEPA: Topology-Aware Self-Supervised Representation Learning for Distributed Tactile Sensors](https://mlsanity.com/paper/2609.24385/) (2026)
- [Self-Supervised Anchoring of Fingertip Sensing to Proprioception and Proactive Actions for Robot Imitation Learning](https://mlsanity.com/paper/2609.29822/) (2026)

## Abstract (arXiv)

We present Sparsh-X, the first multisensory touch representations across four tactile modalities: image, audio, motion, and pressure. Trained on ~1M contact-rich interactions collected with the Digit 360 sensor, Sparsh-X captures complementary touch signals at diverse temporal and spatial scales. By leveraging self-supervised learning, Sparsh-X fuses these modalities into a unified representation that captures physical properties useful for robot manipulation tasks. We study how to effectively integrate real-world touch representations for both imitation learning and tactile adaptation of sim-trained policies, showing that Sparsh-X boosts policy success rates by 63% over an end-to-end model using tactile images and improves robustness by 90% in recovering object states from touch. Finally, we benchmark Sparsh-X ability to make inferences about physical properties, such as object-action identification, material-quantity estimation, and force estimation. Sparsh-X improves accuracy in characterizing physical properties by 48% compared to end-to-end approaches, demonstrating the advantages of multisensory pretraining for capturing features essential for dexterous manipulation.