mlsanityData to 13 Oct 2026

RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer

arXiv 2011.03148 ↗First posted 6 Nov 2020cs.ROcited by 68

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

The success of deep reinforcement learning (RL) and imitation learning (IL) in vision-based robotic manipulation typically hinges on the expense of large scale data collection. With simulation, data to train a policy can be collected efficiently at scale, but the visual gap between sim and real makes deployment in the real world difficult. We introduce RetinaGAN, a generative adversarial network (GAN) approach to adapt simulated images to realistic ones with object-detection consistency. RetinaGAN is trained in an unsupervised manner without task loss dependencies, and preserves general object structure and texture in adapted images. We evaluate our method on three real world tasks: grasping, pushing, and door opening. RetinaGAN improves upon the performance of prior sim-to-real methods for RL-based object instance grasping and continues to be effective even in the limited data regime. When applied to a pushing task in a similar visual domain, RetinaGAN demonstrates transfer with no additional real data requirements. We also show our method bridges the visual gap for a novel door opening task using imitation learning in a new visual domain. Visit the project website at https://retinagan.github.io/

Key facts. Cited by 68 papers in the map (data through 13 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 6 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 4; 2022: 15; 2023: 12; 2024: 14; 2025: 16; 2026: 7.

68 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,321
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,028
  3. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 518
  4. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 485
  5. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 176
  6. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 137
  7. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  8. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 129
  9. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 107
  10. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 103
  11. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  12. 2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 61
38 more
  1. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  2. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 47
  3. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 44
  4. 2021Cross-domain Imitation from Observationscited by 42
  5. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 36
  6. 2023Polybot: Training One Policy Across Robots While Embracing Variabilitycited by 28
  7. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 28
  8. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  9. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 22
  10. 2024Natural Language Can Help Bridge the Sim2Real Gapcited by 21
  11. 2022Sim-to-Real Domain Adaptation for Lane Detection and Classification in Autonomous Drivingcited by 20
  12. 2023Synthetic-to-Real Domain Adaptation for Action Recognition: A Dataset and Baseline Performancescited by 16
  13. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  14. 2025WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perceptioncited by 15
  15. 2022Bayesian Imitation Learning for End-to-End Mobile Manipulationcited by 14
  16. 2025Generative Models in Decision Making: A Surveycited by 14
  17. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 13
  18. 2023What do we learn from a large-scale study of pre-trained visual representations in sim and real environments?cited by 11
  19. 2022Policy Adaptation from Foundation Model Feedbackcited by 10
  20. 2025OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learningcited by 10
  21. 2022Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Datacited by 9
  22. 2025Generalizable Domain Adaptation for Sim-and-Real Policy Co-Trainingcited by 9
  23. 2022Practical Imitation Learning in the Real World via Task Consistency Losscited by 6
  24. 2024Improving Object Detector Training on Synthetic Data by Starting With a Strong Baseline Methodologycited by 6
  25. 2025How Do VLAs Effectively Inherit from VLMs?cited by 6
  26. 2022PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scalecited by 5
  27. 2023Robust Visual Sim-to-Real Transfer for Robotic Manipulationcited by 5
  28. 2024Cross Domain Policy Transfer with Effect Cycle-Consistencycited by 5
  29. 2024Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learningcited by 5
  30. 2023A Framework for Few-Shot Policy Transfer through Observation Mapping and Behavior Cloningcited by 4
  31. 2023DT/MARS-CycleGAN: Improved Object Detection for MARS Phenotyping Robotcited by 4
  32. 2024BodySLAM: A Generalized Monocular Visual SLAM Framework for Surgical Applicationscited by 4
  33. 2022Neural Neural Textures Make Sim2Real Consistentcited by 3
  34. 2022Leveraging Haptic Feedback to Improve Data Quality and Quantity for Deep Imitation Learning Modelscited by 3
  35. 2025LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrationscited by 3
  36. 2022On the Verge of Solving Rocket League using Deep Reinforcement Learning and Sim-to-sim Transfercited by 2
  37. 2023Towards Precise Model-free Robotic Grasping with Sim-to-Real Transfer Learningcited by 2
  38. 2023Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulationcited by 2

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2018Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
  2. 2025Zero-Shot Visual Generalization in Robot Manipulation
  3. 2026Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models
  4. 2023NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis
  5. 2025ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Models
  6. 2026RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulation
  7. 2026RE4: Transformation-aware Imitation of Object Interactions Using Manipulation Modes
  8. 2024ReMoBot: Retrieval-Based Few-Shot Imitation Learning for Mobile Manipulation with Vision Foundation Models