# Attention for Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Control

arXiv: https://arxiv.org/abs/2307.14510 · HTML: https://mlsanity.com/paper/2307.14510/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 26 Jul 2023.

## Builds on (most cited first, 15 of 15)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [Review of Visual Saliency Detection with Comprehensive Information](https://mlsanity.com/paper/1803.03391/) (2018)
- [EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction](https://mlsanity.com/paper/1805.01047/) (2018)
- [Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation](https://mlsanity.com/paper/2106.08796/) (2021)
- [Tactile Gym 2.0: Sim-to-real Deep Reinforcement Learning for Comparing Low-cost High-Resolution Robot Touch](https://mlsanity.com/paper/2207.10763/) (2022)
- [Deep Saliency Prior for Reducing Visual Distraction](https://mlsanity.com/paper/2109.01980/) (2021)
- [Bi-Touch: Bimanual Tactile Manipulation with Sim-to-Real Deep Reinforcement Learning](https://mlsanity.com/paper/2307.06423/) (2023)
- [Soft Biomimetic Optical Tactile Sensing with the TacTip: A Review](https://mlsanity.com/paper/2105.14455/) (2021)
- [Look here! A parametric learning based approach to redirect visual attention](https://mlsanity.com/paper/2008.05413/) (2020)
- [Spatio-temporal Attention Model for Tactile Texture Recognition](https://mlsanity.com/paper/2008.04442/) (2020)
- [DigiTac: A DIGIT-TacTip Hybrid Tactile Sensor for Comparing Low-Cost High-Resolution Robot Touch](https://mlsanity.com/paper/2206.13657/) (2022)
- [Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning](https://mlsanity.com/paper/2204.00117/) (2022)
- [Pose-Based Tactile Servoing: Controlled Soft Touch using Deep Learning](https://mlsanity.com/paper/2012.02504/) (2020)
- [Optimal Deep Learning for Robot Touch](https://mlsanity.com/paper/2003.01916/) (2020)

## Built on by (most cited first, 2 of 2)

- [M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation](https://mlsanity.com/paper/2401.17032/) (2024)
- [SARL: Spatially-Aware Self-Supervised Representation Learning for Visuo-Tactile Perception](https://mlsanity.com/paper/2512.01908/) (2025)

## Abstract (arXiv)

High-resolution tactile sensing can provide accurate information about local contact in contact-rich robotic tasks. However, the deployment of such tasks in unstructured environments remains under-investigated. To improve the robustness of tactile robot control in unstructured environments, we propose and study a new concept: textittactile saliency for robot touch, inspired by the human touch attention mechanism from neuroscience and the visual saliency prediction problem from computer vision. In analogy to visual saliency, this concept involves identifying key information in tactile images captured by a tactile sensor. While visual saliency datasets are commonly annotated by humans, manually labelling tactile images is challenging due to their counterintuitive patterns. To address this challenge, we propose a novel approach comprised of three interrelated networks: 1) a Contact Depth Network (ConDepNet), which generates a contact depth map to localize deformation in a real tactile image that contains target and noise features; 2) a Tactile Saliency Network (TacSalNet), which predicts a tactile saliency map to describe the target areas for an input contact depth map; 3) and a Tactile Noise Generator (TacNGen), which generates noise features to train the TacSalNet. Experimental results in contact pose estimation and edge-following in the presence of distractors showcase the accurate prediction of target features from real tactile images. Overall, our tactile saliency prediction approach gives robust sim-to-real tactile control in environments with unknown distractors. Project page: https://sites.google.com/view/tactile-saliency/.