# Autoregressive Unsupervised Image Segmentation

arXiv: https://arxiv.org/abs/2007.08247 · HTML: https://mlsanity.com/paper/2007.08247/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2020.

## Built on by (most cited first, 30 of 60)

- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [ReCo: Retrieve and Co-segment for Zero-shot Transfer](https://mlsanity.com/paper/2206.07045/) (2022)
- [Large-scale Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2106.03149/) (2021)
- [Discovering Object Masks with Transformers for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2206.06363/) (2022)
- [Unsupervised Semantic Segmentation with Self-supervised Object-centric Representations](https://mlsanity.com/paper/2207.05027/) (2022)
- [Labels4Free: Unsupervised Segmentation using StyleGAN](https://mlsanity.com/paper/2103.14968/) (2021)
- [A Survey on Label-efficient Deep Image Segmentation: Bridging the Gap between Weak Supervision and Dense Prediction](https://mlsanity.com/paper/2207.01223/) (2022)
- [Finding an Unsupervised Image Segmenter in Each of Your Deep Generative Models](https://mlsanity.com/paper/2105.08127/) (2021)
- [Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion](https://mlsanity.com/paper/2205.07844/) (2022)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2112.01515/) (2021)
- [CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.01040/) (2023)
- [GrowSP: Unsupervised Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2305.16404/) (2023)
- [Pixel-Inconsistency Modeling for Image Manipulation Localization](https://mlsanity.com/paper/2310.00234/) (2023)
- [EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2403.01482/) (2024)
- [NamedMask: Distilling Segmenters from Complementary Foundation Models](https://mlsanity.com/paper/2209.11228/) (2022)
- [MOVE: Unsupervised Movable Object Segmentation and Detection](https://mlsanity.com/paper/2210.07920/) (2022)
- [Drive&Segment: Unsupervised Semantic Segmentation of Urban Scenes via Cross-modal Distillation](https://mlsanity.com/paper/2203.11160/) (2022)
- [SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2310.17874/) (2023)
- [Learning Hierarchical Image Segmentation For Recognition and By Recognition](https://mlsanity.com/paper/2210.00314/) (2022)
- [U3DS3: Unsupervised 3D Semantic Scene Segmentation](https://mlsanity.com/paper/2311.06018/) (2023)
- [Visually Consistent Hierarchical Image Classification](https://mlsanity.com/paper/2406.11608/) (2024)
- [Unsupervised Image Segmentation by Mutual Information Maximization and Adversarial Regularization](https://mlsanity.com/paper/2107.00691/) (2021)
- [InfoSeg: Unsupervised Semantic Image Segmentation with Mutual Information Maximization](https://mlsanity.com/paper/2110.03477/) (2021)
- [Fully Self-Supervised Learning for Semantic Segmentation](https://mlsanity.com/paper/2202.11981/) (2022)
- [A Learning-based Framework for Topology-Preserving Segmentation using Quasiconformal Mappings](https://mlsanity.com/paper/2210.03299/) (2022)
- [Rethinking Alignment and Uniformity in Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2211.14513/) (2022)
- [Dense Siamese Network for Dense Unsupervised Learning](https://mlsanity.com/paper/2203.11075/) (2022)
- [Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation](https://mlsanity.com/paper/2204.09847/) (2022)

## Abstract (arXiv)

In this work, we propose a new unsupervised image segmentation approach based on mutual information maximization between different constructed views of the inputs. Taking inspiration from autoregressive generative models that predict the current pixel from past pixels in a raster-scan ordering created with masked convolutions, we propose to use different orderings over the inputs using various forms of masked convolutions to construct different views of the data. For a given input, the model produces a pair of predictions with two valid orderings, and is then trained to maximize the mutual information between the two outputs. These outputs can either be low-dimensional features for representation learning or output clusters corresponding to semantic labels for clustering. While masked convolutions are used during training, in inference, no masking is applied and we fall back to the standard convolution where the model has access to the full input. The proposed method outperforms current state-of-the-art on unsupervised image segmentation. It is simple and easy to implement, and can be extended to other visual tasks and integrated seamlessly into existing unsupervised learning methods requiring different views of the data.