# GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation

arXiv: https://arxiv.org/abs/2112.01036 · HTML: https://mlsanity.com/paper/2112.01036/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 69 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 30 of 54)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location](https://mlsanity.com/paper/2003.07064/) (2020)
- [Self-supervised Single-view 3D Reconstruction via Semantic Consistency](https://mlsanity.com/paper/2003.06473/) (2020)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)
- [Unsupervised Learning of Object Keypoints for Perception and Control](https://mlsanity.com/paper/1906.11883/) (2019)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [Unsupervised Learning of Object Structure and Dynamics from Videos](https://mlsanity.com/paper/1906.07889/) (2019)
- [Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning](https://mlsanity.com/paper/1807.03146/) (2018)
- [Interpretable and Accurate Fine-grained Recognition via Region Grouping](https://mlsanity.com/paper/2005.10411/) (2020)

## Built on by (most cited first, 19 of 19)

- [Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion](https://mlsanity.com/paper/2205.07844/) (2022)
- [MOVE: Unsupervised Movable Object Segmentation and Detection](https://mlsanity.com/paper/2210.07920/) (2022)
- [Recognizing Object by Components with Human Prior Knowledge Enhances Adversarial Robustness of Deep Neural Networks](https://mlsanity.com/paper/2212.01806/) (2022)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [Unsupervised Keypoints from Pretrained Diffusion Models](https://mlsanity.com/paper/2312.00065/) (2023)
- [PaintSeg: Training-free Segmentation via Painting](https://mlsanity.com/paper/2305.19406/) (2023)
- [Few-shot Geometry-Aware Keypoint Localization](https://mlsanity.com/paper/2303.17216/) (2023)
- [Compositional Neural Textures](https://mlsanity.com/paper/2404.12509/) (2024)
- [SC-VAE: Sparse Coding-based Variational Autoencoder with Learned ISTA](https://mlsanity.com/paper/2303.16666/) (2023)
- [Unsupervised Camouflaged Object Segmentation as Domain Adaptation](https://mlsanity.com/paper/2308.04528/) (2023)
- [Pose Prior Learner: Unsupervised Categorical Prior Learning for Pose Estimation](https://mlsanity.com/paper/2410.03858/) (2024)
- [Unsupervised Part Discovery via Dual Representation Alignment](https://mlsanity.com/paper/2408.08108/) (2024)
- [Bridging Neural and Symbolic Representations with Transitional Dictionary Learning](https://mlsanity.com/paper/2308.02000/) (2023)
- [Unsupervised Part Discovery via Descriptor-Based Masked Image Restoration with Optimized Constraints](https://mlsanity.com/paper/2507.11985/) (2025)
- [LatentKeypointGAN: Controlling Images via Latent Keypoints -- Extended Abstract](https://mlsanity.com/paper/2205.03448/) (2022)
- [Depth-Guided Self-Supervised Human Keypoint Detection via Cross-Modal Distillation](https://mlsanity.com/paper/2410.14700/) (2024)
- [Weakly Supervised Object Segmentation by Background Conditional Divergence](https://mlsanity.com/paper/2506.22505/) (2025)
- [Unsupervised Monocular 3D Keypoint Discovery from Multi-View Diffusion Priors](https://mlsanity.com/paper/2507.12336/) (2025)
- [SHED Light on Segmentation for Dense Prediction](https://mlsanity.com/paper/2601.22529/) (2026)

## Abstract (arXiv)

Segmenting an image into its parts is a frequent preprocess for high-level vision tasks such as image editing. However, annotating masks for supervised training is expensive. Weakly-supervised and unsupervised methods exist, but they depend on the comparison of pairs of images, such as from multi-views, frames of videos, and image augmentation, which limits their applicability. To address this, we propose a GAN-based approach that generates images conditioned on latent masks, thereby alleviating full or weak annotations required in previous approaches. We show that such mask-conditioned image generation can be learned faithfully when conditioning the masks in a hierarchical manner on latent keypoints that define the position of parts explicitly. Without requiring supervision of masks or points, this strategy increases robustness to viewpoint and object positions changes. It also lets us generate image-mask pairs for training a segmentation network, which outperforms the state-of-the-art unsupervised segmentation methods on established benchmarks.