# Semantic-aware Grad-GAN for Virtual-to-Real Urban Scene Adaption

arXiv: https://arxiv.org/abs/1801.01726 · HTML: https://mlsanity.com/paper/1801.01726/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jan 2018.

## Built on by (most cited first, 27 of 27)

- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [Enhancing Photorealism Enhancement](https://mlsanity.com/paper/2105.04619/) (2021)
- [Sparse-to-dense Feature Matching: Intra and Inter domain Cross-modal Learning in Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2107.14724/) (2021)
- [Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation](https://mlsanity.com/paper/2109.02618/) (2021)
- [CoMoGAN: continuous model-guided image-to-image translation](https://mlsanity.com/paper/2103.06879/) (2021)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Leveraging Synthetic Data in Object Detection on Unmanned Aerial Vehicles](https://mlsanity.com/paper/2112.12252/) (2021)
- [Unsupervised Domain Adaptation for Semantic Image Segmentation: a Comprehensive Survey](https://mlsanity.com/paper/2112.03241/) (2021)
- [Domain-robust VQA with diverse datasets and methods but no target labels](https://mlsanity.com/paper/2103.15974/) (2021)
- [Physics-informed Guided Disentanglement in Generative Networks](https://mlsanity.com/paper/2107.14229/) (2021)
- [Confidence-based Visual Dispersal for Few-shot Unsupervised Domain Adaptation](https://mlsanity.com/paper/2309.15575/) (2023)
- [ManiFest: Manifold Deformation for Few-shot Image Translation](https://mlsanity.com/paper/2111.13681/) (2021)
- [Content Disentanglement for Semantically Consistent Synthetic-to-Real Domain Adaptation](https://mlsanity.com/paper/2105.08704/) (2021)
- [Segmentation-guided Domain Adaptation and Data Harmonization of Multi-device Retinal Optical Coherence Tomography using Cycle-Consistent Generative Adversarial Networks](https://mlsanity.com/paper/2208.14635/) (2022)
- [Knowing the Distance: Understanding the Gap Between Synthetic and Real Data For Face Parsing](https://mlsanity.com/paper/2303.15219/) (2023)
- [Uncertainty-Aware Model Adaptation for Unsupervised Cross-Domain Object Detection](https://mlsanity.com/paper/2108.12612/) (2021)
- [TridentAdapt: Learning Domain-invariance via Source-Target Confrontation and Self-induced Cross-domain Augmentation](https://mlsanity.com/paper/2111.15300/) (2021)
- [Cooperative Self-Training for Multi-Target Adaptive Semantic Segmentation](https://mlsanity.com/paper/2210.01578/) (2022)
- [A Shared Representation for Photorealistic Driving Simulators](https://mlsanity.com/paper/2112.05134/) (2021)
- [KLIEP-based Density Ratio Estimation for Semantically Consistent Synthetic to Real Images Adaptation in Urban Traffic Scenes](https://mlsanity.com/paper/2105.12549/) (2021)
- [Attention-based Adversarial Appearance Learning of Augmented Pedestrians](https://mlsanity.com/paper/2107.02673/) (2021)
- [Leveraging Local Domains for Image-to-Image Translation](https://mlsanity.com/paper/2109.04468/) (2021)
- [Synthetic-to-real Composite Semantic Segmentation in Additive Manufacturing](https://mlsanity.com/paper/2210.07466/) (2022)
- [Reducing Annotation Effort by Identifying and Labeling Contextually Diverse Classes for Semantic Segmentation Under Domain Shift](https://mlsanity.com/paper/2210.06749/) (2022)
- [SyntStereo2Real: Edge-Aware GAN for Remote Sensing Image-to-Image Translation while Maintaining Stereo Constraint](https://mlsanity.com/paper/2404.09277/) (2024)
- [PartStickers: Generating Parts of Objects for Rapid Prototyping](https://mlsanity.com/paper/2504.05508/) (2025)

## Abstract (arXiv)

Recent advances in vision tasks (e.g., segmentation) highly depend on the availability of large-scale real-world image annotations obtained by cumbersome human labors. Moreover, the perception performance often drops significantly for new scenarios, due to the poor generalization capability of models trained on limited and biased annotations. In this work, we resort to transfer knowledge from automatically rendered scene annotations in virtual-world to facilitate real-world visual tasks. Although virtual-world annotations can be ideally diverse and unlimited, the discrepant data distributions between virtual and real-world make it challenging for knowledge transferring. We thus propose a novel Semantic-aware Grad-GAN (SG-GAN) to perform virtual-to-real domain adaption with the ability of retaining vital semantic information. Beyond the simple holistic color/texture transformation achieved by prior works, SG-GAN successfully personalizes the appearance adaption for each semantic region in order to preserve their key characteristic for better recognition. It presents two main contributions to traditional GANs: 1) a soft gradient-sensitive objective for keeping semantic boundaries; 2) a semantic-aware discriminator for validating the fidelity of personalized adaptions with respect to each semantic region. Qualitative and quantitative experiments demonstrate the superiority of our SG-GAN in scene adaption over state-of-the-art GANs. Further evaluations on semantic segmentation on Cityscapes show using adapted virtual images by SG-GAN dramatically improves segmentation performance than original virtual data. We release our code at https://github.com/Peilun-Li/SG-GAN.