# Simulation of Vision-based Tactile Sensors using Physics based Rendering

arXiv: https://arxiv.org/abs/2012.13184 · HTML: https://mlsanity.com/paper/2012.13184/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2020.

## Built on by (most cited first, 15 of 15)

- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [Vision-Based Manipulators Need to Also See from Their Hands](https://mlsanity.com/paper/2203.12677/) (2022)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Models](https://mlsanity.com/paper/2304.01182/) (2023)
- [TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing](https://mlsanity.com/paper/2311.12602/) (2023)
- [Sensor-Invariant Tactile Representation](https://mlsanity.com/paper/2502.19638/) (2025)
- [Tacchi 2.0: A Low Computational Cost and Comprehensive Dynamic Contact Simulator for Vision-based Tactile Sensors](https://mlsanity.com/paper/2503.09100/) (2025)
- [AI Should Sense Better, Not Just Scale Bigger: Adaptive Sensing as a Paradigm Shift](https://mlsanity.com/paper/2507.07820/) (2025)
- [TouchAI: Exploring human-AI perceptual alignment in touch through language model representations](https://mlsanity.com/paper/2406.06587/) (2024)
- [GelBelt: A Vision-based Tactile Sensor for Continuous Sensing of Large Surfaces](https://mlsanity.com/paper/2501.06263/) (2025)
- [Force-Modulated Visual Policy for Robot-Assisted Dressing with Arm Motions](https://mlsanity.com/paper/2509.12741/) (2025)
- [A Modularized Design Approach for GelSight Family of Vision-based Tactile Sensors](https://mlsanity.com/paper/2504.14739/) (2025)
- [ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation](https://mlsanity.com/paper/2505.20498/) (2025)
- [MultiDiffSense: Diffusion-Based Multi-Modal Visuo-Tactile Image Generation Conditioned on Object Shape and Contact Pose](https://mlsanity.com/paper/2602.19348/) (2026)
- [DOT-Sim: Differentiable Optical Tactile Simulation with Precise Real-to-Sim Physical Calibration](https://mlsanity.com/paper/2604.27367/) (2026)

## Abstract (arXiv)

Tactile sensing has seen a rapid adoption with the advent of vision-based tactile sensors. Vision-based tactile sensors provide high resolution, compact and inexpensive data to perform precise in-hand manipulation and human-robot interaction. However, the simulation of tactile sensors is still a challenge. In this paper, we built the first fully general optical tactile simulation system for a GelSight sensor using physics-based rendering techniques. We propose physically accurate light models and show in-depth analysis of individual components of our simulation pipeline. Our system outperforms previous simulation techniques qualitatively and quantitative on image similarity metrics. Our code and experimental data is open-sourced at https://labs.ri.cmu.edu/robotouch/tactile-optical-simulation/