# Mutual-GAN: Towards Unsupervised Cross-Weather Adaptation with Mutual Information Constraint

arXiv: https://arxiv.org/abs/2106.16000 · HTML: https://mlsanity.com/paper/2106.16000/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [Bidirectional Learning for Domain Adaptation of Semantic Segmentation](https://mlsanity.com/paper/1904.10620/) (2019)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [Learning to Discover Cross-Domain Relations with Generative Adversarial Networks](https://mlsanity.com/paper/1703.05192/) (2017)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [Category Anchor-Guided Unsupervised Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/1910.13049/) (2019)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [Self-Ensembling with GAN-based Data Augmentation for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1909.00589/) (2019)
- [Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime](https://mlsanity.com/paper/1810.02575/) (2018)
- [Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach](https://mlsanity.com/paper/1812.05040/) (2018)
- [Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping](https://mlsanity.com/paper/1809.05852/) (2018)
- [All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation](https://mlsanity.com/paper/1903.12212/) (2019)
- [Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network](https://mlsanity.com/paper/1802.09655/) (2018)
- [ROAD: Reality Oriented Adaptation for Semantic Segmentation of Urban Scenes](https://mlsanity.com/paper/1711.11556/) (2017)
- [Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach](https://mlsanity.com/paper/1810.11547/) (2018)
- [See Clearer at Night: Towards Robust Nighttime Semantic Segmentation through Day-Night Image Conversion](https://mlsanity.com/paper/1908.05868/) (2019)
- [Temporal Coherence for Active Learning in Videos](https://mlsanity.com/paper/1908.11757/) (2019)
- [DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)](https://mlsanity.com/paper/1802.06454/) (2018)
- [Generative Adversarial Networks for Video-to-Video Domain Adaptation](https://mlsanity.com/paper/2004.08058/) (2020)
- [MISO: Mutual Information Loss with Stochastic Style Representations for Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1902.03938/) (2019)

## Built on by (most cited first, 1 of 1)

- [An Analysis for Image-to-Image Translation and Style Transfer](https://mlsanity.com/paper/2408.06000/) (2024)

## Abstract (arXiv)

Convolutional neural network (CNN) have proven its success for semantic segmentation, which is a core task of emerging industrial applications such as autonomous driving. However, most progress in semantic segmentation of urban scenes is reported on standard scenarios, i.e., daytime scenes with favorable illumination conditions. In practical applications, the outdoor weather and illumination are changeable, e.g., cloudy and nighttime, which results in a significant drop of semantic segmentation accuracy of CNN only trained with daytime data. In this paper, we propose a novel generative adversarial network (namely Mutual-GAN) to alleviate the accuracy decline when daytime-trained neural network is applied to videos captured under adverse weather conditions. The proposed Mutual-GAN adopts mutual information constraint to preserve image-objects during cross-weather adaptation, which is an unsolved problem for most unsupervised image-to-image translation approaches (e.g., CycleGAN). The proposed Mutual-GAN is evaluated on two publicly available driving video datasets (i.e., CamVid and SYNTHIA). The experimental results demonstrate that our Mutual-GAN can yield visually plausible translated images and significantly improve the semantic segmentation accuracy of daytime-trained deep learning network while processing videos under challenging weathers.