# Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation

arXiv: https://arxiv.org/abs/2101.02680 · HTML: https://mlsanity.com/paper/2101.02680/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 7 Jan 2021.

## Builds on (most cited first, 16 of 16)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019)
- [Manipulation by Feel: Touch-Based Control with Deep Predictive Models](https://mlsanity.com/paper/1903.04128/) (2019)
- [Tactile Dexterity: Manipulation Primitives with Tactile Feedback](https://mlsanity.com/paper/2002.03236/) (2020)
- [Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering](https://mlsanity.com/paper/2012.05205/) (2020)
- [Location Sensitive Deep Convolutional Neural Networks for Segmentation of White Matter Hyperintensities](https://mlsanity.com/paper/1610.04834/) (2016)
- [Ground truth force distribution for learning-based tactile sensing: a finite element approach](https://mlsanity.com/paper/1909.04000/) (2019)
- [Sim-to-Real Transfer for Optical Tactile Sensing](https://mlsanity.com/paper/2004.00136/) (2020)
- [Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing](https://mlsanity.com/paper/2003.02640/) (2020)
- [Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard](https://mlsanity.com/paper/2008.02646/) (2020)
- [Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions](https://mlsanity.com/paper/2012.11295/) (2020)
- [Elastic Interaction of Particles for Robotic Tactile Simulation](https://mlsanity.com/paper/2011.11528/) (2020)

## Built on by (most cited first, 14 of 14)

- [Rotating without Seeing: Towards In-hand Dexterity through Touch](https://mlsanity.com/paper/2303.10880/) (2023)
- [3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing](https://mlsanity.com/paper/2410.24091/) (2024)
- [Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding](https://mlsanity.com/paper/2501.04693/) (2025)
- [The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning](https://mlsanity.com/paper/2311.00924/) (2023)
- [Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation](https://mlsanity.com/paper/2106.08796/) (2021)
- [Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning](https://mlsanity.com/paper/2403.12170/) (2024)
- [FlexiTac: A Low-Cost, Open-Source, Scalable Tactile Sensing Solution for Robotic Systems](https://mlsanity.com/paper/2604.28156/) (2026)
- [LeTac-MPC: Learning Model Predictive Control for Tactile-reactive Grasping](https://mlsanity.com/paper/2403.04934/) (2024)
- [On the Importance of Tactile Sensing for Imitation Learning: A Case Study on Robotic Match Lighting](https://mlsanity.com/paper/2504.13618/) (2025)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)
- [Leveraging distributed contact force measurements for slip detection: a physics-based approach enabled by a data-driven tactile sensor](https://mlsanity.com/paper/2109.11504/) (2021)
- [Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach](https://mlsanity.com/paper/2302.13423/) (2023)
- [Towards Transferring Tactile-based Continuous Force Control Policies from Simulation to Robot](https://mlsanity.com/paper/2311.07245/) (2023)
- [Learning Gentle Grasping from Human-Free Force Control Demonstration](https://mlsanity.com/paper/2409.10371/) (2024)

## Abstract (arXiv)

This paper aims to show that robots equipped with a vision-based tactile sensor can perform dynamic manipulation tasks without prior knowledge of all the physical attributes of the objects to be manipulated. For this purpose, a robotic system is presented that is able to swing up poles of different masses, radii and lengths, to an angle of 180 degrees, while relying solely on the feedback provided by the tactile sensor. This is achieved by developing a novel simulator that accurately models the interaction of a pole with the soft sensor. A feedback policy that is conditioned on a sensory observation history, and which has no prior knowledge of the physical features of the pole, is then learned in the aforementioned simulation. When evaluated on the physical system, the policy is able to swing up a wide range of poles that differ significantly in their physical attributes without further adaptation. To the authors' knowledge, this is the first work where a feedback policy from high-dimensional tactile observations is used to control the swing-up manipulation of poles in closed-loop.