# InfoSeg: Unsupervised Semantic Image Segmentation with Mutual Information Maximization

arXiv: https://arxiv.org/abs/2110.03477 · HTML: https://mlsanity.com/paper/2110.03477/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [SegSort: Segmentation by Discriminative Sorting of Segments](https://mlsanity.com/paper/1910.06962/) (2019)
- [Autoregressive Unsupervised Image Segmentation](https://mlsanity.com/paper/2007.08247/) (2020)
- [Learning visual groups from co-occurrences in space and time](https://mlsanity.com/paper/1511.06811/) (2015)
- [Unsupervised Image Segmentation by Mutual Information Maximization and Adversarial Regularization](https://mlsanity.com/paper/2107.00691/) (2021)

## Built on by (most cited first, 11 of 11)

- [CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2306.04300/) (2023)
- [TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2112.01515/) (2021)
- [SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2310.17874/) (2023)
- [Object-centric Learning with Cyclic Walks between Parts and Whole](https://mlsanity.com/paper/2302.08023/) (2023)
- [Rethinking Alignment and Uniformity in Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2211.14513/) (2022)
- [Unsupervised Image Semantic Segmentation through Superpixels and Graph Neural Networks](https://mlsanity.com/paper/2210.11810/) (2022)
- [Removing supervision in semantic segmentation with local-global matching and area balancing](https://mlsanity.com/paper/2303.17410/) (2023)
- [DatUS^2: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformer](https://mlsanity.com/paper/2401.12820/) (2024)
- [AI-based Automatic Segmentation of Prostate on Multi-modality Images: A Review](https://mlsanity.com/paper/2407.06612/) (2024)
- [Unsupervised Monocular Road Segmentation for Autonomous Driving via Scene Geometry](https://mlsanity.com/paper/2510.16790/) (2025)
- [High-Resolution Image Reconstruction with Unsupervised Learning and Noisy Data Applied to Ion-Beam Dynamics for Particle Accelerators](https://mlsanity.com/paper/2603.06689/) (2026)

## Abstract (arXiv)

We propose a novel method for unsupervised semantic image segmentation based on mutual information maximization between local and global high-level image features. The core idea of our work is to leverage recent progress in self-supervised image representation learning. Representation learning methods compute a single high-level feature capturing an entire image. In contrast, we compute multiple high-level features, each capturing image segments of one particular semantic class. To this end, we propose a novel two-step learning procedure comprising a segmentation and a mutual information maximization step. In the first step, we segment images based on local and global features. In the second step, we maximize the mutual information between local features and high-level features of their respective class. For training, we provide solely unlabeled images and start from random network initialization. For quantitative and qualitative evaluation, we use established benchmarks, and COCO-Persons, whereby we introduce the latter in this paper as a challenging novel benchmark. InfoSeg significantly outperforms the current state-of-the-art, e.g., we achieve a relative increase of 26% in the Pixel Accuracy metric on the COCO-Stuff dataset.