# From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor

arXiv: https://arxiv.org/abs/1812.02941 · HTML: https://mlsanity.com/paper/1812.02941/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2018.

## Built on by (most cited first, 12 of 12)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger](https://mlsanity.com/paper/2106.08851/) (2021)
- [SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation](https://mlsanity.com/paper/2101.11812/) (2021)
- [Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation](https://mlsanity.com/paper/2106.08796/) (2021)
- [Tac2Pose: Tactile Object Pose Estimation from the First Touch](https://mlsanity.com/paper/2204.11701/) (2022)
- [Vision-Guided Active Tactile Perception for Crack Detection and Reconstruction](https://mlsanity.com/paper/2105.06325/) (2021)
- [Learning to Synthesize Volumetric Meshes from Vision-based Tactile Imprints](https://mlsanity.com/paper/2203.15155/) (2022)
- [Tactile Image-to-Image Disentanglement of Contact Geometry from Motion-Induced Shear](https://mlsanity.com/paper/2109.03615/) (2021)
- [Robot Active Neural Sensing and Planning in Unknown Cluttered Environments](https://mlsanity.com/paper/2208.11079/) (2022)
- [Semi-Supervised Disentanglement of Tactile Contact~Geometry from Sliding-Induced Shear](https://mlsanity.com/paper/2208.12500/) (2022)
- [Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition](https://mlsanity.com/paper/2311.11287/) (2023)
- [More with LESS -- Local Scene Representations for Tactile Imaging](https://mlsanity.com/paper/2606.14344/) (2026)

## Abstract (arXiv)

Deep learning has the potential to have the impact on robot touch that it has had on robot vision. Optical tactile sensors act as a bridge between the subjects by allowing techniques from vision to be applied to touch. In this paper, we apply deep learning to an optical biomimetic tactile sensor, the TacTip, which images an array of papillae (pins) inside its sensing surface analogous to structures within human skin. Our main result is that the application of a deep CNN can give reliable edge perception and thus a robust policy for planning contact points to move around object contours. Robustness is demonstrated over several irregular and compliant objects with both tapping and continuous sliding, using a model trained only by tapping onto a disk. These results relied on using techniques to encourage generalization to tasks beyond which the model was trained. We expect this is a generic problem in practical applications of tactile sensing that deep learning will solve. A video demonstrating the approach can be found at https://www.youtube.com/watch?v=QHrGsG9AHts