# Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projections

arXiv: https://arxiv.org/abs/2103.16747 · HTML: https://mlsanity.com/paper/2103.16747/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2021.

## Built on by (most cited first, 28 of 28)

- [Factory: Fast Contact for Robotic Assembly](https://mlsanity.com/paper/2205.03532/) (2022)
- [Visuo-Tactile Transformers for Manipulation](https://mlsanity.com/paper/2210.00121/) (2022)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World](https://mlsanity.com/paper/2401.08577/) (2024)
- [AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch](https://mlsanity.com/paper/2405.07391/) (2024)
- [Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation](https://mlsanity.com/paper/2106.08796/) (2021)
- [A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI](https://mlsanity.com/paper/2505.01458/) (2025)
- [TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing](https://mlsanity.com/paper/2311.12602/) (2023)
- [Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensing](https://mlsanity.com/paper/2407.07885/) (2024)
- [Visuo-Haptic Object Perception for Robots: An Overview](https://mlsanity.com/paper/2203.11544/) (2022)
- [AcTExplore: Active Tactile Exploration of Unknown Objects](https://mlsanity.com/paper/2310.08745/) (2023)
- [Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning](https://mlsanity.com/paper/2403.12170/) (2024)
- [OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation](https://mlsanity.com/paper/2110.00704/) (2021)
- [DiSECt: A Differentiable Simulator for Parameter Inference and Control in Robotic Cutting](https://mlsanity.com/paper/2203.10263/) (2022)
- [FlexiTac: A Low-Cost, Open-Source, Scalable Tactile Sensing Solution for Robotic Systems](https://mlsanity.com/paper/2604.28156/) (2026)
- [TwinTac: A Wide-Range, Highly Sensitive Tactile Sensor with Real-to-Sim Digital Twin Sensor Model](https://mlsanity.com/paper/2509.10063/) (2025)
- [HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2603.00446/) (2026)
- [Learning to Synthesize Volumetric Meshes from Vision-based Tactile Imprints](https://mlsanity.com/paper/2203.15155/) (2022)
- [Integrated Object Deformation and Contact Patch Estimation from Visuo-Tactile Feedback](https://mlsanity.com/paper/2305.14470/) (2023)
- [Telextiles: End-to-end Remote Transmission of Fabric Tactile Sensation](https://mlsanity.com/paper/2405.03363/) (2024)
- [Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning](https://mlsanity.com/paper/2606.11767/) (2026)
- [Optimizing BioTac Simulation for Realistic Tactile Perception](https://mlsanity.com/paper/2404.10425/) (2024)
- [Transferring Tactile Data Across Sensors](https://mlsanity.com/paper/2410.14310/) (2024)
- [ACROSS: A Deformation-Based Cross-Modal Representation for Robotic Tactile Perception](https://mlsanity.com/paper/2411.08533/) (2024)
- [Towards Transferring Tactile-based Continuous Force Control Policies from Simulation to Robot](https://mlsanity.com/paper/2311.07245/) (2023)
- [Estimating Material Properties of Interacting Objects Using Sum-GP-UCB](https://mlsanity.com/paper/2310.11749/) (2023)
- [SPLIT: Separating Physical-Contact via Latent Arithmetic in Image-Based Tactile Sensors](https://mlsanity.com/paper/2604.24449/) (2026)
- [Reduced-order Neural Modeling with Differentiable Simulation for High-Detail Tactile Perception](https://mlsanity.com/paper/2605.05053/) (2026)

## Abstract (arXiv)

Tactile sensing is critical for robotic grasping and manipulation of objects under visual occlusion. However, in contrast to simulations of robot arms and cameras, current simulations of tactile sensors have limited accuracy, speed, and utility. In this work, we develop an efficient 3D finite element method (FEM) model of the SynTouch BioTac sensor using an open-access, GPU-based robotics simulator. Our simulations closely reproduce results from an experimentally-validated model in an industry-standard, CPU-based simulator, but at 75x the speed. We then learn latent representations for simulated BioTac deformations and real-world electrical output through self-supervision, as well as projections between the latent spaces using a small supervised dataset. Using these learned latent projections, we accurately synthesize real-world BioTac electrical output and estimate contact patches, both for unseen contact interactions. This work contributes an efficient, freely-accessible FEM model of the BioTac and comprises one of the first efforts to combine self-supervision, cross-modal transfer, and sim-to-real transfer for tactile sensors.