# RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real

arXiv: https://arxiv.org/abs/2006.09001 · HTML: https://mlsanity.com/paper/2006.09001/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 30 of 91)

- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Scaling Robot Learning with Semantically Imagined Experience](https://mlsanity.com/paper/2302.11550/) (2023)
- [RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking](https://mlsanity.com/paper/2309.01918/) (2023)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control](https://mlsanity.com/paper/2503.14492/) (2025)
- [When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?](https://mlsanity.com/paper/2204.05618/) (2022)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)
- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)
- [Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes](https://mlsanity.com/paper/2110.06192/) (2021)
- [SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting](https://mlsanity.com/paper/2409.10161/) (2024)
- [Advances and Challenges in Meta-Learning: A Technical Review](https://mlsanity.com/paper/2307.04722/) (2023)
- [When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning](https://mlsanity.com/paper/2206.13464/) (2022)
- [Holo-Dex: Teaching Dexterity with Immersive Mixed Reality](https://mlsanity.com/paper/2210.06463/) (2022)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/2503.24361/) (2025)
- [SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies](https://mlsanity.com/paper/2106.09678/) (2021)
- [Cross-domain Imitation from Observations](https://mlsanity.com/paper/2105.10037/) (2021)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [Sketch Your Own GAN](https://mlsanity.com/paper/2108.02774/) (2021)
- [Self-Improving Embodied Foundation Models](https://mlsanity.com/paper/2509.15155/) (2025)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation](https://mlsanity.com/paper/2402.14795/) (2024)
- [A Dataset Perspective on Offline Reinforcement Learning](https://mlsanity.com/paper/2111.04714/) (2021)
- [World-Gymnast: Training Robots with Reinforcement Learning in a World Model](https://mlsanity.com/paper/2602.02454/) (2026)
- [Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery](https://mlsanity.com/paper/2406.06092/) (2024)
- [A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions](https://mlsanity.com/paper/2109.03540/) (2021)
- [Natural Language Can Help Bridge the Sim2Real Gap](https://mlsanity.com/paper/2405.10020/) (2024)
- [Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation](https://mlsanity.com/paper/2106.08796/) (2021)
- [Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs](https://mlsanity.com/paper/2311.02847/) (2023)

## Abstract (arXiv)

Deep neural network based reinforcement learning (RL) can learn appropriate visual representations for complex tasks like vision-based robotic grasping without the need for manually engineering or prior learning a perception system. However, data for RL is collected via running an agent in the desired environment, and for applications like robotics, running a robot in the real world may be extremely costly and time consuming. Simulated training offers an appealing alternative, but ensuring that policies trained in simulation can transfer effectively into the real world requires additional machinery. Simulations may not match reality, and typically bridging the simulation-to-reality gap requires domain knowledge and task-specific engineering. We can automate this process by employing generative models to translate simulated images into realistic ones. However, this sort of translation is typically task-agnostic, in that the translated images may not preserve all features that are relevant to the task. In this paper, we introduce the RL-scene consistency loss for image translation, which ensures that the translation operation is invariant with respect to the Q-values associated with the image. This allows us to learn a task-aware translation. Incorporating this loss into unsupervised domain translation, we obtain RL-CycleGAN, a new approach for simulation-to-real-world transfer for reinforcement learning. In evaluations of RL-CycleGAN on two vision-based robotics grasping tasks, we show that RL-CycleGAN offers a substantial improvement over a number of prior methods for sim-to-real transfer, attaining excellent real-world performance with only a modest number of real-world observations.