# Sem-GAN: Semantically-Consistent Image-to-Image Translation

arXiv: https://arxiv.org/abs/1807.04409 · HTML: https://mlsanity.com/paper/1807.04409/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2018.

## Built on by (most cited first, 26 of 26)

- [Neural Optimal Transport](https://mlsanity.com/paper/2201.12220/) (2022)
- [Enhancing Photorealism Enhancement](https://mlsanity.com/paper/2105.04619/) (2021)
- [CoMoGAN: continuous model-guided image-to-image translation](https://mlsanity.com/paper/2103.06879/) (2021)
- [Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization](https://mlsanity.com/paper/2102.06315/) (2021)
- [On the Road to Online Adaptation for Semantic Image Segmentation](https://mlsanity.com/paper/2203.16195/) (2022)
- [Cycle and Semantic Consistent Adversarial Domain Adaptation for Reducing Simulation-to-Real Domain Shift in LiDAR Bird's Eye View](https://mlsanity.com/paper/2104.11021/) (2021)
- [Physics-informed Guided Disentanglement in Generative Networks](https://mlsanity.com/paper/2107.14229/) (2021)
- [RaidaR: A Rich Annotated Image Dataset of Rainy Street Scenes](https://mlsanity.com/paper/2104.04606/) (2021)
- [ManiFest: Manifold Deformation for Few-shot Image Translation](https://mlsanity.com/paper/2111.13681/) (2021)
- [Implicit Multi-Spectral Transformer: An Lightweight and Effective Visible to Infrared Image Translation Model](https://mlsanity.com/paper/2404.07072/) (2024)
- [Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution](https://mlsanity.com/paper/2507.14367/) (2025)
- [Bridging Day and Night: Target-Class Hallucination Suppression in Unpaired Image Translation](https://mlsanity.com/paper/2602.15383/) (2026)
- [StyleFlow For Content-Fixed Image to Image Translation](https://mlsanity.com/paper/2207.01909/) (2022)
- [Self-adversarial Multi-scale Contrastive Learning for Semantic Segmentation of Thermal Facial Images](https://mlsanity.com/paper/2209.10700/) (2022)
- [A Shared Representation for Photorealistic Driving Simulators](https://mlsanity.com/paper/2112.05134/) (2021)
- [Nighttime Thermal Infrared Image Colorization with Feedback-based Object Appearance Learning](https://mlsanity.com/paper/2310.15688/) (2023)
- [S-CycleGAN: Semantic Segmentation Enhanced CT-Ultrasound Image-to-Image Translation for Robotic Ultrasonography](https://mlsanity.com/paper/2406.01191/) (2024)
- [Leveraging Local Domains for Image-to-Image Translation](https://mlsanity.com/paper/2109.04468/) (2021)
- [Memory-Guided Collaborative Attention for Nighttime Thermal Infrared Image Colorization](https://mlsanity.com/paper/2208.02960/) (2022)
- [Noise Consistency Regularization for Improved Subject-Driven Image Synthesis](https://mlsanity.com/paper/2506.06483/) (2025)
- [SDA-GAN: Unsupervised Image Translation Using Spectral Domain Attention-Guided Generative Adversarial Network](https://mlsanity.com/paper/2110.02873/) (2021)
- [Semantic Map Injected GAN Training for Image-to-Image Translation](https://mlsanity.com/paper/2112.01845/) (2021)
- [Interpreting CNN Predictions using Conditional Generative Adversarial Networks](https://mlsanity.com/paper/2301.08067/) (2023)
- [A Survey on Recent Random Walk-based Methods for Embedding Knowledge Graphs](https://mlsanity.com/paper/2406.07402/) (2024)
- [Sun Off, Lights On: Photorealistic Monocular Nighttime Simulation for Robust Semantic Perception](https://mlsanity.com/paper/2407.20336/) (2024)
- [Enhanced Unsupervised Image-to-Image Translation Using Contrastive Learning and Histogram of Oriented Gradients](https://mlsanity.com/paper/2409.16042/) (2024)

## Abstract (arXiv)

Unpaired image-to-image translation is the problem of mapping an image in the source domain to one in the target domain, without requiring corresponding image pairs. To ensure the translated images are realistically plausible, recent works, such as Cycle-GAN, demands this mapping to be invertible. While, this requirement demonstrates promising results when the domains are unimodal, its performance is unpredictable in a multi-modal scenario such as in an image segmentation task. This is because, invertibility does not necessarily enforce semantic correctness. To this end, we present a semantically-consistent GAN framework, dubbed Sem-GAN, in which the semantics are defined by the class identities of image segments in the source domain as produced by a semantic segmentation algorithm. Our proposed framework includes consistency constraints on the translation task that, together with the GAN loss and the cycle-constraints, enforces that the images when translated will inherit the appearances of the target domain, while (approximately) maintaining their identities from the source domain. We present experiments on several image-to-image translation tasks and demonstrate that Sem-GAN improves the quality of the translated images significantly, sometimes by more than 20% on the FCN score. Further, we show that semantic segmentation models, trained with synthetic images translated via Sem-GAN, leads to significantly better segmentation results than other variants.