# Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut

arXiv: https://arxiv.org/abs/2202.11539 · HTML: https://mlsanity.com/paper/2202.11539/

## Key facts

- Cited by 167 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 47 of its 81 references that are in the map.
- First posted on arXiv 23 Feb 2022.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017)
- [Unbiased Teacher for Semi-Supervised Object Detection](https://mlsanity.com/paper/2102.09480/) (2021)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localization](https://mlsanity.com/paper/1704.04232/) (2017)
- [Adversarial Complementary Learning for Weakly Supervised Object Localization](https://mlsanity.com/paper/1804.06962/) (2018)
- [Attention-based Dropout Layer for Weakly Supervised Object Localization](https://mlsanity.com/paper/1908.10028/) (2019)
- [MST: Masked Self-Supervised Transformer for Visual Representation](https://mlsanity.com/paper/2106.05656/) (2021)
- [Evaluating Weakly Supervised Object Localization Methods Right](https://mlsanity.com/paper/2001.07437/) (2020)
- [Salient Object Detection: A Discriminative Regional Feature Integration Approach](https://mlsanity.com/paper/1410.5926/) (2014)
- [The Fast Bilateral Solver](https://mlsanity.com/paper/1511.03296/) (2015)
- [Weakly Supervised Object Localization and Detection: A Survey](https://mlsanity.com/paper/2104.07918/) (2021)

## Built on by (most cited first, 30 of 167)

- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations](https://mlsanity.com/paper/2209.03494/) (2022)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [ReCo: Retrieve and Co-segment for Zero-shot Transfer](https://mlsanity.com/paper/2206.07045/) (2022)
- [FoundPose: Unseen Object Pose Estimation with Foundation Features](https://mlsanity.com/paper/2311.18809/) (2023)
- [CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation](https://mlsanity.com/paper/2312.12359/) (2023)
- [Unsupervised Object Localization: Observing the Background to Discover Objects](https://mlsanity.com/paper/2212.07834/) (2022)
- [LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Grounding](https://mlsanity.com/paper/2303.09665/) (2023)
- [Diffusion Models for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2306.09316/) (2023)
- [Improving Object-centric Learning with Query Optimization](https://mlsanity.com/paper/2210.08990/) (2022)
- [Unsupervised Salient Object Detection with Spectral Cluster Voting](https://mlsanity.com/paper/2203.12614/) (2022)
- [Unsupervised Semantic Segmentation with Self-supervised Object-centric Representations](https://mlsanity.com/paper/2207.05027/) (2022)
- [DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery](https://mlsanity.com/paper/2303.09813/) (2023)
- [Self-supervised Object-Centric Learning for Videos](https://mlsanity.com/paper/2310.06907/) (2023)
- [FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models](https://mlsanity.com/paper/2303.12786/) (2023)
- [Teaching Matters: Investigating the Role of Supervision in Vision Transformers](https://mlsanity.com/paper/2212.03862/) (2022)
- [Three Pillars improving Vision Foundation Model Distillation for Lidar](https://mlsanity.com/paper/2310.17504/) (2023)
- [Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2411.19331/) (2024)
- [EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models](https://mlsanity.com/paper/2401.11739/) (2024)
- [Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion](https://mlsanity.com/paper/2205.07844/) (2022)
- [ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2210.05944/) (2022)
- [Optimizing Relevance Maps of Vision Transformers Improves Robustness](https://mlsanity.com/paper/2206.01161/) (2022)
- [Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models](https://mlsanity.com/paper/2210.15138/) (2022)
- [CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free](https://mlsanity.com/paper/2309.14289/) (2023)
- [In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2408.04961/) (2024)
- [Neural Systematic Binder](https://mlsanity.com/paper/2211.01177/) (2022)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentation](https://mlsanity.com/paper/2411.15869/) (2024)

## Abstract (arXiv)

Transformers trained with self-supervised learning using self-distillation loss (DINO) have been shown to produce attention maps that highlight salient foreground objects. In this paper, we demonstrate a graph-based approach that uses the self-supervised transformer features to discover an object from an image. Visual tokens are viewed as nodes in a weighted graph with edges representing a connectivity score based on the similarity of tokens. Foreground objects can then be segmented using a normalized graph-cut to group self-similar regions. We solve the graph-cut problem using spectral clustering with generalized eigen-decomposition and show that the second smallest eigenvector provides a cutting solution since its absolute value indicates the likelihood that a token belongs to a foreground object. Despite its simplicity, this approach significantly boosts the performance of unsupervised object discovery: we improve over the recent state of the art LOST by a margin of 6.9%, 8.1%, and 8.1% respectively on the VOC07, VOC12, and COCO20K. The performance can be further improved by adding a second stage class-agnostic detector (CAD). Our proposed method can be easily extended to unsupervised saliency detection and weakly supervised object detection. For unsupervised saliency detection, we improve IoU for 4.9%, 5.2%, 12.9% on ECSSD, DUTS, DUT-OMRON respectively compared to previous state of the art. For weakly supervised object detection, we achieve competitive performance on CUB and ImageNet.