# Synthetic-to-Real Domain Adaptation using Contrastive Unpaired Translation

arXiv: https://arxiv.org/abs/2203.09454 · HTML: https://mlsanity.com/paper/2203.09454/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 14 of 14)

- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [DLOW: Domain Flow for Adaptation and Generalization](https://mlsanity.com/paper/1812.05418/) (2018)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)
- [HomebrewedDB: RGB-D Dataset for 6D Pose Estimation of 3D Objects](https://mlsanity.com/paper/1904.03167/) (2019)
- [Cartman: The low-cost Cartesian Manipulator that won the Amazon Robotics Challenge](https://mlsanity.com/paper/1709.06283/) (2017)
- [Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing](https://mlsanity.com/paper/1810.02977/) (2018)
- [Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics](https://mlsanity.com/paper/2005.05659/) (2020)
- [GeneSIS-RT: Generating Synthetic Images for training Secondary Real-world Tasks](https://mlsanity.com/paper/1710.04280/) (2017)

## Built on by (most cited first, 10 of 10)

- [DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception](https://mlsanity.com/paper/2403.13304/) (2024)
- [Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery](https://mlsanity.com/paper/2406.06092/) (2024)
- [Generating Diverse Agricultural Data for Vision-Based Farming Applications](https://mlsanity.com/paper/2403.18351/) (2024)
- [Beyond Real Data: Synthetic Data through the Lens of Regularization](https://mlsanity.com/paper/2510.08095/) (2025)
- [Synthetic-to-real Composite Semantic Segmentation in Additive Manufacturing](https://mlsanity.com/paper/2210.07466/) (2022)
- [Learning from SAM: Harnessing a Foundation Model for Sim2Real Adaptation by Regularization](https://mlsanity.com/paper/2309.15562/) (2023)
- [A Scalable Pipeline Combining Procedural 3D Graphics and Guided Diffusion for Photorealistic Synthetic Training Data Generation in White Button Mushroom Segmentation](https://mlsanity.com/paper/2512.08747/) (2025)
- [WoodScape Motion Segmentation for Autonomous Driving -- CVPR 2023 OmniCV Workshop Challenge](https://mlsanity.com/paper/2401.00910/) (2023)
- [IntrinsicReal: Adapting IntrinsicAnything from Synthetic to Real Objects](https://mlsanity.com/paper/2509.00777/) (2025)
- [FD-DB: Frequency-Decoupled Dual-Branch Network for Unpaired Synthetic-to-Real Domain Translation](https://mlsanity.com/paper/2602.09476/) (2026)

## Abstract (arXiv)

The usefulness of deep learning models in robotics is largely dependent on the availability of training data. Manual annotation of training data is often infeasible. Synthetic data is a viable alternative, but suffers from domain gap. We propose a multi-step method to obtain training data without manual annotation effort: From 3D object meshes, we generate images using a modern synthesis pipeline. We utilize a state-of-the-art image-to-image translation method to adapt the synthetic images to the real domain, minimizing the domain gap in a learned manner. The translation network is trained from unpaired images, i.e. just requires an un-annotated collection of real images. The generated and refined images can then be used to train deep learning models for a particular task. We also propose and evaluate extensions to the translation method that further increase performance, such as patch-based training, which shortens training time and increases global consistency. We evaluate our method and demonstrate its effectiveness on two robotic datasets. We finally give insight into the learned refinement operations.