# AptSim2Real: Approximately-Paired Sim-to-Real Image Translation

arXiv: https://arxiv.org/abs/2303.12704 · HTML: https://mlsanity.com/paper/2303.12704/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [Coupled Generative Adversarial Networks](https://mlsanity.com/paper/1606.07536/) (2016)
- [Projected GANs Converge Faster](https://mlsanity.com/paper/2111.01007/) (2021)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [One-Sided Unsupervised Domain Mapping](https://mlsanity.com/paper/1706.00826/) (2017)
- [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016)
- [The Unusual Effectiveness of Averaging in GAN Training](https://mlsanity.com/paper/1806.04498/) (2018)
- [InstaFormer: Instance-Aware Image-to-Image Translation with Transformer](https://mlsanity.com/paper/2203.16248/) (2022)
- [Transformation Consistency Regularization- A Semi-Supervised Paradigm for Image-to-Image Translation](https://mlsanity.com/paper/2007.07867/) (2020)
- [Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts](https://mlsanity.com/paper/1612.00215/) (2016)
- [Unsupervised Image-to-Image Translation with Generative Prior](https://mlsanity.com/paper/2204.03641/) (2022)
- [Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models](https://mlsanity.com/paper/2110.02891/) (2021)

## Built on by (most cited first, 1 of 1)

- [RenderBench: Benchmarking Render-to-Real Video Transfer with Reconstructed Digital Twins](https://mlsanity.com/paper/2610.08684/) (2026)

## Abstract (arXiv)

Advancements in graphics technology has increased the use of simulated data for training machine learning models. However, the simulated data often differs from real-world data, creating a distribution gap that can decrease the efficacy of models trained on simulation data in real-world applications. To mitigate this gap, sim-to-real domain transfer modifies simulated images to better match real-world data, enabling the effective use of simulation data in model training. Sim-to-real transfer utilizes image translation methods, which are divided into two main categories: paired and unpaired image-to-image translation. Paired image translation requires a perfect pixel match, making it difficult to apply in practice due to the lack of pixel-wise correspondence between simulation and real-world data. Unpaired image translation, while more suitable for sim-to-real transfer, is still challenging to learn for complex natural scenes. To address these challenges, we propose a third category: approximately-paired sim-to-real translation, where the source and target images do not need to be exactly paired. Our approximately-paired method, AptSim2Real, exploits the fact that simulators can generate scenes loosely resembling real-world scenes in terms of lighting, environment, and composition. Our novel training strategy results in significant qualitative and quantitative improvements, with up to a 24% improvement in FID score compared to the state-of-the-art unpaired image-translation methods.