# Learning Neural Eigenfunctions for Unsupervised Semantic Segmentation

arXiv: https://arxiv.org/abs/2304.02841 · HTML: https://mlsanity.com/paper/2304.02841/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 6 Apr 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [Simple Does It: Weakly Supervised Instance and Semantic Segmentation](https://mlsanity.com/paper/1603.07485/) (2016)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)
- [PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering](https://mlsanity.com/paper/2103.17070/) (2021)
- [Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization](https://mlsanity.com/paper/2205.07839/) (2022)
- [ReCo: Retrieve and Co-segment for Zero-shot Transfer](https://mlsanity.com/paper/2206.07045/) (2022)
- [SegSort: Segmentation by Discriminative Sorting of Segments](https://mlsanity.com/paper/1910.06962/) (2019)
- [Unsupervised Object Segmentation by Redrawing](https://mlsanity.com/paper/1905.13539/) (2019)
- [Deep Spectral Clustering using Dual Autoencoder Network](https://mlsanity.com/paper/1904.13113/) (2019)
- [Self-Supervised Visual Representation Learning from Hierarchical Grouping](https://mlsanity.com/paper/2012.03044/) (2020)
- [Autoregressive Unsupervised Image Segmentation](https://mlsanity.com/paper/2007.08247/) (2020)
- [Labels4Free: Unsupervised Segmentation using StyleGAN](https://mlsanity.com/paper/2103.14968/) (2021)
- [Object Segmentation Without Labels with Large-Scale Generative Models](https://mlsanity.com/paper/2006.04988/) (2020)
- [Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning](https://mlsanity.com/paper/2105.00957/) (2021)
- [Spectral Inference Networks: Unifying Deep and Spectral Learning](https://mlsanity.com/paper/1806.02215/) (2018)
- [A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains](https://mlsanity.com/paper/1912.11186/) (2019)

## Built on by (most cited first, 4 of 4)

- [EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2403.01482/) (2024)
- [ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2408.06747/) (2024)
- [AMNCutter: Affinity-Attention-Guided Multi-View Normalized Cutter for Unsupervised Surgical Instrument Segmentation](https://mlsanity.com/paper/2411.03695/) (2024)
- [Zero-Shot Edge Detection with SCESAME: Spectral Clustering-based Ensemble for Segment Anything Model Estimation](https://mlsanity.com/paper/2308.13779/) (2023)

## Abstract (arXiv)

Unsupervised semantic segmentation is a long-standing challenge in computer vision with great significance. Spectral clustering is a theoretically grounded solution to it where the spectral embeddings for pixels are computed to construct distinct clusters. Despite recent progress in enhancing spectral clustering with powerful pre-trained models, current approaches still suffer from inefficiencies in spectral decomposition and inflexibility in applying them to the test data. This work addresses these issues by casting spectral clustering as a parametric approach that employs neural network-based eigenfunctions to produce spectral embeddings. The outputs of the neural eigenfunctions are further restricted to discrete vectors that indicate clustering assignments directly. As a result, an end-to-end NN-based paradigm of spectral clustering emerges. In practice, the neural eigenfunctions are lightweight and take the features from pre-trained models as inputs, improving training efficiency and unleashing the potential of pre-trained models for dense prediction. We conduct extensive empirical studies to validate the effectiveness of our approach and observe significant performance gains over competitive baselines on Pascal Context, Cityscapes, and ADE20K benchmarks.