# Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensor

arXiv: https://arxiv.org/abs/1704.03955 · HTML: https://mlsanity.com/paper/1704.03955/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2017.

## Built on by (most cited first, 19 of 19)

- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger](https://mlsanity.com/paper/2106.08851/) (2021)
- [Generating Visual Scenes from Touch](https://mlsanity.com/paper/2309.15117/) (2023)
- [DenseTact: Optical Tactile Sensor for Dense Shape Reconstruction](https://mlsanity.com/paper/2201.01367/) (2022)
- [Dynamic Modeling of Hand-Object Interactions via Tactile Sensing](https://mlsanity.com/paper/2109.04378/) (2021)
- [Controllable Visual-Tactile Synthesis](https://mlsanity.com/paper/2305.03051/) (2023)
- [Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Losses](https://mlsanity.com/paper/2107.05468/) (2021)
- [Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Dataset](https://mlsanity.com/paper/2403.09813/) (2024)
- [LeTac-MPC: Learning Model Predictive Control for Tactile-reactive Grasping](https://mlsanity.com/paper/2403.04934/) (2024)
- [Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis](https://mlsanity.com/paper/2411.03315/) (2024)
- [Predicting Physical Object Properties from Video](https://mlsanity.com/paper/2206.00930/) (2022)
- [Visuo-Tactile Zero-Shot Object Recognition with Vision-Language Model](https://mlsanity.com/paper/2409.09276/) (2024)
- [Tube Diffusion Policy: Reactive Visual-Tactile Policy Learning for Contact-rich Manipulation](https://mlsanity.com/paper/2604.23609/) (2026)
- [Towards Design and Development of an ArUco Markers-Based Quantitative Surface Tactile Sensor](https://mlsanity.com/paper/2310.08398/) (2023)
- [Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges](https://mlsanity.com/paper/2505.08453/) (2025)
- [Investigating Active Sampling for Hardness Classification with Vision-Based Tactile Sensors](https://mlsanity.com/paper/2505.13231/) (2025)
- [ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation](https://mlsanity.com/paper/2505.20498/) (2025)
- [Advances in Compliance Detection: Novel Models Using Vision-Based Tactile Sensors](https://mlsanity.com/paper/2506.14980/) (2025)

## Abstract (arXiv)

Hardness is among the most important attributes of an object that humans learn about through touch. However, approaches for robots to estimate hardness are limited, due to the lack of information provided by current tactile sensors. In this work, we address these limitations by introducing a novel method for hardness estimation, based on the GelSight tactile sensor, and the method does not require accurate control of contact conditions or the shape of objects. A GelSight has a soft contact interface, and provides high resolution tactile images of contact geometry, as well as contact force and slip conditions. In this paper, we try to use the sensor to measure hardness of objects with multiple shapes, under a loosely controlled contact condition. The contact is made manually or by a robot hand, while the force and trajectory are unknown and uneven. We analyze the data using a deep constitutional (and recurrent) neural network. Experiments show that the neural net model can estimate the hardness of objects with different shapes and hardness ranging from 8 to 87 in Shore 00 scale.