mlsanityData to 14 Oct 2026

Robust Visual Sim-to-Real Transfer for Robotic Manipulation

arXiv 2307.15320 ↗First posted 28 Jul 2023cs.RO cs.AI cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Learning visuomotor policies in simulation is much safer and cheaper than in the real world. However, due to discrepancies between the simulated and real data, simulator-trained policies often fail when transferred to real robots. One common approach to bridge the visual sim-to-real domain gap is domain randomization (DR). While previous work mainly evaluates DR for disembodied tasks, such as pose estimation and object detection, here we systematically explore visual domain randomization methods and benchmark them on a rich set of challenging robotic manipulation tasks. In particular, we propose an off-line proxy task of cube localization to select DR parameters for texture randomization, lighting randomization, variations of object colors and camera parameters. Notably, we demonstrate that DR parameters have similar impact on our off-line proxy task and on-line policies. We, hence, use off-line optimized DR parameters to train visuomotor policies in simulation and directly apply such policies to a real robot. Our approach achieves 93% success rate on average when tested on a diverse set of challenging manipulation tasks. Moreover, we evaluate the robustness of policies to visual variations in real scenes and show that our simulator-trained policies outperform policies learned using real but limited data. Code, simulation environment, real robot datasets and trained models are available at https://www.di.ens.fr/willow/research/robust_s2r/.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 28 Jul 2023.

Builds on

34 of 44 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  5. 2019Solving Rubik's Cube with a Robot Handcited by 734
  6. 2018Learning Dexterous In-Hand Manipulationcited by 687
  7. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  8. 2021Implicit Behavioral Cloningcited by 452
  9. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  10. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  11. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 289
  12. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
22 more
  1. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  2. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  3. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 174
  4. 2019Active Domain Randomizationcited by 170
  5. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 151
  6. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 149
  7. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 144
  8. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 131
  9. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  10. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 107
  11. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 91
  12. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 84
  13. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 71
  14. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 70
  15. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  16. 2021Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFingercited by 38
  17. 2022Locomotion Policy Guided Traversability Learning using Volumetric Representations of Complex Environmentscited by 28
  18. 2020Perspectives on Sim2Real Transfer for Robotics: A Summary of the R:SS 2020 Workshopcited by 25
  19. 2020Crossing The Gap: A Deep Dive into Zero-Shot Sim-to-Real Transfer for Dynamicscited by 21
  20. 2019Learning to Augment Synthetic Images for Sim2Real Policy Transfercited by 16
  21. 2020Benchmarking Domain Randomisation for Visual Sim-to-Real Transfercited by 15
  22. 2021DROID: Minimizing the Reality Gap using Single-Shot Human Demonstrationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  2. 2025Robot Motion Planning using One-Step Diffusion with Noise-Optimized Approximate Motionscited by 3
  3. 2026Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasetscited by 2
  4. 2025Learning to Grasp Anything by Playing with Random Toyscited by 0
  5. 2026ObstaDiff: Generalizable Diffusion Policy Learning via Obstacle-aware Representationscited by 0