# Extracting Semantic Knowledge from GANs with Unsupervised Learning

arXiv: https://arxiv.org/abs/2211.16710 · HTML: https://mlsanity.com/paper/2211.16710/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 77 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs](https://mlsanity.com/paper/2005.09635/) (2020)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [On the "steerability" of generative adversarial networks](https://mlsanity.com/paper/1907.07171/) (2019)
- [Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks](https://mlsanity.com/paper/1506.05751/) (2015)

## Built on by (most cited first, 5 of 5)

- [A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends](https://mlsanity.com/paper/2402.15490/) (2024)
- [Personalized Face Inpainting with Diffusion Models by Parallel Visual Attention](https://mlsanity.com/paper/2312.03556/) (2023)
- [RmGPT: A Foundation Model with Generative Pre-trained Transformer for Fault Diagnosis and Prognosis in Rotating Machinery](https://mlsanity.com/paper/2409.17604/) (2024)
- [Enhancing Semantic Communication with Deep Generative Models -- An ICASSP Special Session Overview](https://mlsanity.com/paper/2309.02478/) (2023)
- [Linking in Style: Understanding learned features in deep learning models](https://mlsanity.com/paper/2409.16865/) (2024)

## Abstract (arXiv)

Recently, unsupervised learning has made impressive progress on various tasks. Despite the dominance of discriminative models, increasing attention is drawn to representations learned by generative models and in particular, Generative Adversarial Networks (GANs). Previous works on the interpretation of GANs reveal that GANs encode semantics in feature maps in a linearly separable form. In this work, we further find that GAN's features can be well clustered with the linear separability assumption. We propose a novel clustering algorithm, named KLiSH, which leverages the linear separability to cluster GAN's features. KLiSH succeeds in extracting fine-grained semantics of GANs trained on datasets of various objects, e.g., car, portrait, animals, and so on. With KLiSH, we can sample images from GANs along with their segmentation masks and synthesize paired image-segmentation datasets. Using the synthesized datasets, we enable two downstream applications. First, we train semantic segmentation networks on these datasets and test them on real images, realizing unsupervised semantic segmentation. Second, we train image-to-image translation networks on the synthesized datasets, enabling semantic-conditional image synthesis without human annotations.