# Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions

arXiv: https://arxiv.org/abs/2012.11295 · HTML: https://mlsanity.com/paper/2012.11295/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2020.

## Built on by (most cited first, 6 of 6)

- [HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation](https://mlsanity.com/paper/2403.10506/) (2024)
- [The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning](https://mlsanity.com/paper/2311.00924/) (2023)
- [Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation](https://mlsanity.com/paper/2101.02680/) (2021)
- [Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis](https://mlsanity.com/paper/2411.03315/) (2024)
- [Leveraging distributed contact force measurements for slip detection: a physics-based approach enabled by a data-driven tactile sensor](https://mlsanity.com/paper/2109.11504/) (2021)
- [Tactile Curiosity Drives Robot Interaction](https://mlsanity.com/paper/2609.40134/) (2026)

## Abstract (arXiv)

The images captured by vision-based tactile sensors carry information about high-resolution tactile fields, such as the distribution of the contact forces applied to their soft sensing surface. However, extracting the information encoded in the images is challenging and often addressed with learning-based approaches, which generally require a large amount of training data. This article proposes a strategy to generate tactile images in simulation for a vision-based tactile sensor based on an internal camera that tracks the motion of spherical particles within a soft material. The deformation of the material is simulated in a finite element environment under a diverse set of contact conditions, and spherical particles are projected to a simulated image. Features extracted from the images are mapped to the 3D contact force distribution, with the ground truth also obtained via finite-element simulations, with an artificial neural network that is therefore entirely trained on synthetic data avoiding the need for real-world data collection. The resulting model exhibits high accuracy when evaluated on real-world tactile images, is transferable across multiple tactile sensors without further training, and is suitable for efficient real-time inference.