mlsanityData to 7 Oct 2026

Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation

arXiv 2106.08796 ↗First posted 16 Jun 2021cs.RO cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Simulation has recently become key for deep reinforcement learning to safely and efficiently acquire general and complex control policies from visual and proprioceptive inputs. Tactile information is not usually considered despite its direct relation to environment interaction. In this work, we present a suite of simulated environments tailored towards tactile robotics and reinforcement learning. A simple and fast method of simulating optical tactile sensors is provided, where high-resolution contact geometry is represented as depth images. Proximal Policy Optimisation (PPO) is used to learn successful policies across all considered tasks. A data-driven approach enables translation of the current state of a real tactile sensor to corresponding simulated depth images. This policy is implemented within a real-time control loop on a physical robot to demonstrate zero-shot sim-to-real policy transfer on several physically-interactive tasks requiring a sense of touch.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

34 of 47 references.

References · 34

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  6. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  7. 2019Solving Rubik's Cube with a Robot Handcited by 699
  8. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  11. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  12. 2020Reinforcement Learning with Augmented Datacited by 452
22 more
  1. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 248
  2. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  3. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  4. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 164
  5. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  6. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  7. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 77
  8. 2020TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensorscited by 59
  9. 2019Value constrained model-free continuous controlcited by 34
  10. 2020OmniTact: A Multi-Directional High Resolution Touch Sensorcited by 27
  11. 2021Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projectionscited by 27
  12. 2020Soft-Bubble grippers for robust and perceptive manipulationcited by 18
  13. 2021Generation of GelSight Tactile Images for Sim2Real Learningcited by 14
  14. 2021Soft Biomimetic Optical Tactile Sensing with the TacTip: A Reviewcited by 14
  15. 2021Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulationcited by 13
  16. 2018From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensorcited by 12
  17. 2020Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensingcited by 7
  18. 2020Sim-to-Real Transfer for Optical Tactile Sensingcited by 7
  19. 2020Goal-Driven Robotic Pushing Using Tactile and Proprioceptive Feedbackcited by 7
  20. 2020Pose-Based Tactile Servoing: Controlled Soft Touch using Deep Learningcited by 6
  21. 2021Interpreting and Predicting Tactile Signals for the SynTouch BioTaccited by 6
  22. 2020Optimal Deep Learning for Robot Touchcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 4; 2025: 2; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  2. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  3. 20243D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensingcited by 67
  4. 2024AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touchcited by 18
  5. 2023Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Modelscited by 17
  6. 2023TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensingcited by 15
  7. 2024Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfacescited by 12
  8. 2022One-Shot Domain-Adaptive Imitation Learning via Progressive Learningcited by 5
  9. 2023Tactile-Filter: Interactive Tactile Perception for Part Matingcited by 4
  10. 2025On the Importance of Tactile Sensing for Imitation Learning: A Case Study on Robotic Match Lightingcited by 4
  11. 2026FlexiTac: A Low-Cost, Open-Source, Scalable Tactile Sensing Solution for Robotic Systemscited by 4
  12. 2026Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learningcited by 3
5 more
  1. 2023Attention for Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Controlcited by 2
  2. 2025Tactile MNIST: Benchmarking Active Tactile Perceptioncited by 2
  3. 2023Towards Transferring Tactile-based Continuous Force Control Policies from Simulation to Robotcited by 1
  4. 2026TacForcing: Streaming Action Generation with Execution-Time Tactile Feedbackcited by 1
  5. 2026Reduced-order Neural Modeling with Differentiable Simulation for High-Detail Tactile Perceptioncited by 0