# Towards Learning to Detect and Predict Contact Events on Vision-based Tactile Sensors

arXiv: https://arxiv.org/abs/1910.03973 · HTML: https://mlsanity.com/paper/1910.03973/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 8 of 8)

- [Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing](https://mlsanity.com/paper/2210.09297/) (2022)
- [Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning](https://mlsanity.com/paper/2403.12170/) (2024)
- [Action Conditioned Tactile Prediction: case study on slip prediction](https://mlsanity.com/paper/2205.09430/) (2022)
- [Leveraging distributed contact force measurements for slip detection: a physics-based approach enabled by a data-driven tactile sensor](https://mlsanity.com/paper/2109.11504/) (2021)
- [A Framework for Multisensory Foresight for Embodied Agents](https://mlsanity.com/paper/2109.07561/) (2021)
- [Learning to Detect Slip with Barometric Tactile Sensors and a Temporal Convolutional Neural Network](https://mlsanity.com/paper/2202.09549/) (2022)
- [Under Pressure: Learning to Detect Slip with Barometric Tactile Sensors](https://mlsanity.com/paper/2103.13460/) (2021)
- [Multi-Modal World Model for Physical Robot Interactions: Simultaneous Visual and Tactile Predictions for Enhanced Accuracy](https://mlsanity.com/paper/2304.11193/) (2023)

## Abstract (arXiv)

In essence, successful grasp boils down to correct responses to multiple contact events between fingertips and objects. In most scenarios, tactile sensing is adequate to distinguish contact events. Due to the nature of high dimensionality of tactile information, classifying spatiotemporal tactile signals using conventional model-based methods is difficult. In this work, we propose to predict and classify tactile signal using deep learning methods, seeking to enhance the adaptability of the robotic grasp system to external event changes that may lead to grasping failure. We develop a deep learning framework and collect 6650 tactile image sequences with a vision-based tactile sensor, and the neural network is integrated into a contact-event-based robotic grasping system. In grasping experiments, we achieved 52% increase in terms of object lifting success rate with contact detection, significantly higher robustness under unexpected loads with slip prediction compared with open-loop grasps, demonstrating that integration of the proposed framework into robotic grasping system substantially improves picking success rate and capability to withstand external disturbances.