# Elastic Tactile Simulation Towards Tactile-Visual Perception

arXiv: https://arxiv.org/abs/2108.05013 · HTML: https://mlsanity.com/paper/2108.05013/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 11 Aug 2021.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Self-Supervised Model Adaptation for Multimodal Semantic Segmentation](https://mlsanity.com/paper/1808.03833/) (2018)
- [3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2007.03778/) (2020)
- [Manipulation by Feel: Touch-Based Control with Deep Predictive Models](https://mlsanity.com/paper/1903.04128/) (2019)
- [3D Shape Perception from Monocular Vision, Touch, and Shape Priors](https://mlsanity.com/paper/1808.03247/) (2018)
- [Multi-Modal Geometric Learning for Grasping and Manipulation](https://mlsanity.com/paper/1803.07671/) (2018)
- [Sim-to-Real Transfer for Optical Tactile Sensing](https://mlsanity.com/paper/2004.00136/) (2020)
- [Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing](https://mlsanity.com/paper/2003.02640/) (2020)

## Built on by (most cited first, 5 of 5)

- [Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play](https://mlsanity.com/paper/2303.12076/) (2023)
- [Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2408.01366/) (2024)
- [Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing](https://mlsanity.com/paper/2210.04026/) (2022)
- [Metarobotics for Industry and Society: Vision, Technologies, and Opportunities](https://mlsanity.com/paper/2404.00797/) (2024)
- [A Multi-modal Fusion Network for Terrain Perception Based on Illumination Aware](https://mlsanity.com/paper/2505.11066/) (2025)

## Abstract (arXiv)

Tactile sensing plays an important role in robotic perception and manipulation tasks. To overcome the real-world limitations of data collection, simulating tactile response in a virtual environment comes as a desirable direction of robotic research. In this paper, we propose Elastic Interaction of Particles (EIP) for tactile simulation. Most existing works model the tactile sensor as a rigid multi-body, which is incapable of reflecting the elastic property of the tactile sensor as well as characterizing the fine-grained physical interaction between the two objects. By contrast, EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. With the tactile simulation by EIP, we further propose a tactile-visual perception network that enables information fusion between tactile data and visual images. The perception network is based on a global-to-local fusion mechanism where multi-scale tactile features are aggregated to the corresponding local region of the visual modality with the guidance of tactile positions and directions. The fusion method exhibits superiority regarding the 3D geometric reconstruction task.