# Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis

arXiv: https://arxiv.org/abs/2411.03315 · HTML: https://mlsanity.com/paper/2411.03315/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 51 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger](https://mlsanity.com/paper/1803.00628/) (2018)
- [Slip Detection with Combined Tactile and Visual Information](https://mlsanity.com/paper/1802.10153/) (2018)
- [Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensor](https://mlsanity.com/paper/1704.03955/) (2017)
- [Dense Tactile Force Distribution Estimation using GelSlim and inverse FEM](https://mlsanity.com/paper/1810.04621/) (2018)
- [Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation](https://mlsanity.com/paper/2312.01236/) (2023)
- [Tac3D: A Novel Vision-based Tactile Sensor for Measuring Forces Distribution and Estimating Friction Coefficient Distribution](https://mlsanity.com/paper/2202.06211/) (2022)
- [A soft thumb-sized vision-based sensor with accurate all-round force perception](https://mlsanity.com/paper/2111.05934/) (2021)
- [Ground truth force distribution for learning-based tactile sensing: a finite element approach](https://mlsanity.com/paper/1909.04000/) (2019)
- [Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing](https://mlsanity.com/paper/2003.02640/) (2020)
- [Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions](https://mlsanity.com/paper/2012.11295/) (2020)
- [Minsight: A Fingertip-Sized Vision-Based Tactile Sensor for Robotic Manipulation](https://mlsanity.com/paper/2304.10990/) (2023)
- [Placing by Touching: An empirical study on the importance of tactile sensing for precise object placing](https://mlsanity.com/paper/2210.02054/) (2022)
- [Visual Tactile Sensor Based Force Estimation for Position-Force Teleoperation](https://mlsanity.com/paper/2212.13007/) (2022)
- [Deep Domain Adaptation Regression for Force Calibration of Optical Tactile Sensors](https://mlsanity.com/paper/2407.14380/) (2024)

## Built on by (most cited first, 4 of 4)

- [TaF-VLA: Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulation](https://mlsanity.com/paper/2601.20321/) (2026)
- [X-Capture: An Open-Source Portable Device for Multi-Sensory Learning](https://mlsanity.com/paper/2504.02318/) (2025)
- [Monocular Reconstruction of Neural Tactile Fields](https://mlsanity.com/paper/2602.12508/) (2026)
- [Zero-Shot Transfer of Force Map Estimation Across GelSight Mini Sensors](https://mlsanity.com/paper/2608.18240/) (2026)

## Abstract (arXiv)

Contact-rich manipulation remains a major challenge in robotics. Optical tactile sensors like GelSight Mini offer a low-cost solution for contact sensing by capturing soft-body deformations of the silicone gel. However, accurately inferring shear and normal force distributions from these gel deformations has yet to be fully addressed. In this work, we propose a machine learning approach using a U-net architecture to predict force distributions directly from the sensor's raw images. Our model, trained on force distributions inferred from acfea, demonstrates promising accuracy in predicting normal and shear force distributions for the commercially available GelSight Mini sensor. It also shows potential for generalization across indenters, sensors of the same type, and for enabling real-time application. The codebase, dataset and models are open-sourced and available at https://feats-ai.github.io .