# Grasp Stability Prediction with Sim-to-Real Transfer from Tactile Sensing

arXiv: https://arxiv.org/abs/2208.02885 · HTML: https://mlsanity.com/paper/2208.02885/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2022.

## Built on by (most cited first, 8 of 8)

- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training](https://mlsanity.com/paper/2401.12024/) (2024)
- [Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Models](https://mlsanity.com/paper/2304.01182/) (2023)
- [FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors](https://mlsanity.com/paper/2410.02048/) (2024)
- [Adapting World Models with Latent-State Dynamics Residuals](https://mlsanity.com/paper/2504.02252/) (2025)
- [LeTac-MPC: Learning Model Predictive Control for Tactile-reactive Grasping](https://mlsanity.com/paper/2403.04934/) (2024)
- [Learning to Double Guess: An Active Perception Approach for Estimating the Center of Mass of Arbitrary Objects](https://mlsanity.com/paper/2502.02663/) (2025)
- [A Modularized Design Approach for GelSight Family of Vision-based Tactile Sensors](https://mlsanity.com/paper/2504.14739/) (2025)

## Abstract (arXiv)

Robot simulation has been an essential tool for data-driven manipulation tasks. However, most existing simulation frameworks lack either efficient and accurate models of physical interactions with tactile sensors or realistic tactile simulation. This makes the sim-to-real transfer for tactile-based manipulation tasks still challenging. In this work, we integrate simulation of robot dynamics and vision-based tactile sensors by modeling the physics of contact. This contact model uses simulated contact forces at the robot's end-effector to inform the generation of realistic tactile outputs. To eliminate the sim-to-real transfer gap, we calibrate our physics simulator of robot dynamics, contact model, and tactile optical simulator with real-world data, and then we demonstrate the effectiveness of our system on a zero-shot sim-to-real grasp stability prediction task where we achieve an average accuracy of 90.7% on various objects. Experiments reveal the potential of applying our simulation framework to more complicated manipulation tasks. We open-source our simulation framework at https://github.com/CMURoboTouch/Taxim/tree/taxim-robot.