mlsanityData to 7 Oct 2026

Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks

arXiv 1812.07252 ↗First posted 18 Dec 2018cs.RO cs.CV cs.LGcited by 171

Not on the map. Why?

Abstract (arXiv)

Real world data, especially in the domain of robotics, is notoriously costly to collect. One way to circumvent this can be to leverage the power of simulation to produce large amounts of labelled data. However, training models on simulated images does not readily transfer to real-world ones. Using domain adaptation methods to cross this "reality gap" requires a large amount of unlabelled real-world data, whilst domain randomization alone can waste modeling power. In this paper, we present Randomized-to-Canonical Adaptation Networks (RCANs), a novel approach to crossing the visual reality gap that uses no real-world data. Our method learns to translate randomized rendered images into their equivalent non-randomized, canonical versions. This in turn allows for real images to also be translated into canonical sim images. We demonstrate the effectiveness of this sim-to-real approach by training a vision-based closed-loop grasping reinforcement learning agent in simulation, and then transferring it to the real world to attain 70% zero-shot grasp success on unseen objects, a result that almost doubles the success of learning the same task directly on domain randomization alone. Additionally, by joint finetuning in the real-world with only 5,000 real-world grasps, our method achieves 91%, attaining comparable performance to a state-of-the-art system trained with 580,000 real-world grasps, resulting in a reduction of real-world data by more than 99%.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 50; 2022: 35; 2023: 29; 2024: 22; 2025: 21; 2026: 14.

171 papers.

Sort:

By area:

  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  2. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  3. 2022Kubric: A scalable dataset generatorcited by 373
  4. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  5. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  6. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  7. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  8. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 107
  9. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  10. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  11. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 78
  12. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
38 more
  1. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  2. 2024EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learningcited by 72
  3. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  4. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  5. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  6. 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splattingcited by 63
  7. 2021Robot Learning from Randomized Simulations: A Reviewcited by 56
  8. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 48
  9. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  10. 2021MetaCorrection: Domain-aware Meta Loss Correction for Unsupervised Domain Adaptation in Semantic Segmentationcited by 43
  11. 2021Understanding Continual Learning Settings with Data Distribution Drift Analysiscited by 38
  12. 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 38
  13. 2023EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulationcited by 37
  14. 2023Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learningcited by 36
  15. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 36
  16. 2021Single-view robot pose and joint angle estimation via render & comparecited by 33
  17. 2021ToAlign: Task-oriented Alignment for Unsupervised Domain Adaptationcited by 33
  18. 2021Deep Learning on Monocular Object Pose Detection and Tracking: A Comprehensive Overviewcited by 31
  19. 2022RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimationcited by 29
  20. 2022CPPF: Towards Robust Category-Level 9D Pose Estimation in the Wildcited by 28
  21. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  22. 2023The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learningcited by 24
  23. 2022Learning High-DOF Reaching-and-Grasping via Dynamic Representation of Gripper-Object Interactioncited by 23
  24. 2023Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfercited by 23
  25. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 22
  26. 2024Natural Language Can Help Bridge the Sim2Real Gapcited by 21
  27. 2021DemoGrasp: Few-Shot Learning for Robotic Grasping with Human Demonstrationcited by 20
  28. 2022Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin Pickingcited by 20
  29. 2025ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstrationcited by 20
  30. 2021Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learningcited by 18
  31. 2023TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfercited by 18
  32. 2021Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translationcited by 17
  33. 2024Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RLcited by 17
  34. 2021InsertionNet -- A Scalable Solution for Insertioncited by 15
  35. 2023Synthetic-to-Real Domain Adaptation for Action Recognition: A Dataset and Baseline Performancescited by 15
  36. 2022Coarse-to-Fine Q-attention with Learned Path Rankingcited by 13
  37. 2022Learning Deep Sensorimotor Policies for Vision-based Autonomous Drone Racingcited by 13
  38. 2021Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Groundingcited by 12

50 most cited of 171 All 171 (JSON)