# A soft thumb-sized vision-based sensor with accurate all-round force perception

arXiv: https://arxiv.org/abs/2111.05934 · HTML: https://mlsanity.com/paper/2111.05934/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 62 references that are in the map.
- First posted on arXiv 10 Nov 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger](https://mlsanity.com/paper/1803.00628/) (2018)
- [TriFinger: An Open-Source Robot for Learning Dexterity](https://mlsanity.com/paper/2008.03596/) (2020)
- [Event-Driven Visual-Tactile Sensing and Learning for Robots](https://mlsanity.com/paper/2009.07083/) (2020)
- [Improved GelSight Tactile Sensor for Measuring Geometry and Slip](https://mlsanity.com/paper/1708.00922/) (2017)
- [OmniTact: A Multi-Directional High Resolution Touch Sensor](https://mlsanity.com/paper/2003.06965/) (2020)
- [Dense Tactile Force Distribution Estimation using GelSlim and inverse FEM](https://mlsanity.com/paper/1810.04621/) (2018)
- [GelTip: A Finger-shaped Optical Tactile Sensor for Robotic Manipulation](https://mlsanity.com/paper/2008.05404/) (2020)
- [A Sensorized Multicurved Robot Finger with Data-driven Touch Sensing via Overlapping Light Signals](https://mlsanity.com/paper/2004.00685/) (2020)
- [Soft, Round, High Resolution Tactile Fingertip Sensors for Dexterous Robotic Manipulation](https://mlsanity.com/paper/2005.09068/) (2020)
- [Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing](https://mlsanity.com/paper/2003.02640/) (2020)
- [Interpreting and Predicting Tactile Signals via a Physics-Based and Data-Driven Framework](https://mlsanity.com/paper/2006.03777/) (2020)
- [Modeling Engagement in Long-Term, In-Home Socially Assistive Robot Interventions for Children with Autism Spectrum Disorders](https://mlsanity.com/paper/2002.02453/) (2020)
- [Design of a Fully Actuated Robotic Hand With Multiple Gelsight Tactile Sensors](https://mlsanity.com/paper/2002.02474/) (2020)

## Built on by (most cited first, 10 of 10)

- [Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks](https://mlsanity.com/paper/2406.13640/) (2024)
- [The Essential Role of Causality in Foundation World Models for Embodied AI](https://mlsanity.com/paper/2402.06665/) (2024)
- [GelSight Svelte: A Human Finger-shaped Single-camera Tactile Robot Finger with Large Sensing Coverage and Proprioceptive Sensing](https://mlsanity.com/paper/2309.10885/) (2023)
- [TactoFind: A Tactile Only System for Object Retrieval](https://mlsanity.com/paper/2303.13482/) (2023)
- [Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis](https://mlsanity.com/paper/2411.03315/) (2024)
- [NLiPsCalib: An Efficient Calibration Framework for High-Fidelity 3D Reconstruction of Curved Visuotactile Sensors](https://mlsanity.com/paper/2603.09319/) (2026)
- [Enhance Vision-based Tactile Sensors via Dynamic Illumination and Image Fusion](https://mlsanity.com/paper/2504.00017/) (2025)
- [Proprioceptive Learning with Soft Polyhedral Networks](https://mlsanity.com/paper/2308.08538/) (2023)
- [The Role of Tactile Sensing for Learning Reach and Grasp](https://mlsanity.com/paper/2502.20367/) (2025)
- [PROD: Palpative Reconstruction of Deformable Objects through Elastostatic Signed Distance Functions](https://mlsanity.com/paper/2508.12554/) (2025)

## Abstract (arXiv)

Vision-based haptic sensors have emerged as a promising approach to robotic touch due to affordable high-resolution cameras and successful computer-vision techniques. However, their physical design and the information they provide do not yet meet the requirements of real applications. We present a robust, soft, low-cost, vision-based, thumb-sized 3D haptic sensor named Insight: it continually provides a directional force-distribution map over its entire conical sensing surface. Constructed around an internal monocular camera, the sensor has only a single layer of elastomer over-molded on a stiff frame to guarantee sensitivity, robustness, and soft contact. Furthermore, Insight is the first system to combine photometric stereo and structured light using a collimator to detect the 3D deformation of its easily replaceable flexible outer shell. The force information is inferred by a deep neural network that maps images to the spatial distribution of 3D contact force (normal and shear). Insight has an overall spatial resolution of 0.4 mm, force magnitude accuracy around 0.03 N, and force direction accuracy around 5 degrees over a range of 0.03--2 N for numerous distinct contacts with varying contact area. The presented hardware and software design concepts can be transferred to a wide variety of robot parts.