mlsanityData to 7 Oct 2026

Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization

arXiv 2205.07839 ↗First posted 16 May 2022cs.CV cs.AIcited by 150

Not on the map. Why?

Abstract (arXiv)

Unsupervised localization and segmentation are long-standing computer vision challenges that involve decomposing an image into semantically-meaningful segments without any labeled data. These tasks are particularly interesting in an unsupervised setting due to the difficulty and cost of obtaining dense image annotations, but existing unsupervised approaches struggle with complex scenes containing multiple objects. Differently from existing methods, which are purely based on deep learning, we take inspiration from traditional spectral segmentation methods by reframing image decomposition as a graph partitioning problem. Specifically, we examine the eigenvectors of the Laplacian of a feature affinity matrix from self-supervised networks. We find that these eigenvectors already decompose an image into meaningful segments, and can be readily used to localize objects in a scene. Furthermore, by clustering the features associated with these segments across a dataset, we can obtain well-delineated, nameable regions, i.e. semantic segmentations. Experiments on complex datasets (Pascal VOC, MS-COCO) demonstrate that our simple spectral method outperforms the state-of-the-art in unsupervised localization and segmentation by a significant margin. Furthermore, our method can be readily used for a variety of complex image editing tasks, such as background removal and compositing.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Builds on 90 of its 121 references that are in the map. First posted on arXiv 16 May 2022.

Builds on

90 of 121 references.

References · 90

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  13. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  14. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  15. 2019Contrastive Multiview Codingcited by 1,297
  16. 2016Colorful Image Colorizationcited by 1,113
  17. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  18. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  19. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  20. 2020Object-Centric Learning with Slot Attentioncited by 865
  21. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  22. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  23. 2021Going deeper with Image Transformerscited by 759
  24. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  25. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  26. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  27. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  28. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  29. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  30. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  31. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  32. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  33. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  34. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  35. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  36. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  37. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  38. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 22; 2023: 62; 2024: 41; 2025: 16; 2026: 9.

150 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  2. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 219
  3. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  4. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  5. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  6. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  7. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 115
  8. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  9. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  10. 2022TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cutcited by 111
  11. 2023FoundPose: Unseen Object Pose Estimation with Foundation Featurescited by 109
  12. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
38 more
  1. 2022Unsupervised Object Localization: Observing the Background to Discover Objectscited by 66
  2. 2023LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Groundingcited by 64
  3. 2024DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Videocited by 64
  4. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  5. 2022Diffusion Adversarial Representation Learning for Self-supervised Vessel Segmentationcited by 55
  6. 2022Unsupervised Semantic Segmentation with Self-supervised Object-centric Representationscited by 54
  7. 2023DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discoverycited by 53
  8. 2023Self-supervised Object-Centric Learning for Videoscited by 49
  9. 2023SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformerscited by 49
  10. 2023Spectrum-guided Multi-granularity Referring Video Object Segmentationcited by 48
  11. 2023FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Modelscited by 47
  12. 2022Exploring Target Representations for Masked Autoencoderscited by 46
  13. 2022ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentationcited by 42
  14. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 42
  15. 2022Open-vocabulary Semantic Segmentation with Frozen Vision-Language Modelscited by 39
  16. 2023CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Freecited by 39
  17. 2022Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understandingcited by 37
  18. 2022Self-Guided Diffusion Modelscited by 36
  19. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  20. 2023CLIP-S4: Language-Guided Self-Supervised Semantic Segmentationcited by 34
  21. 2023Neural Congealing: Aligning Images to a Joint Semantic Atlascited by 30
  22. 2024EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentationcited by 27
  23. 2022DeepCut: Unsupervised Segmentation using Graph Neural Networks Clusteringcited by 25
  24. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 23
  25. 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentationcited by 23
  26. 2022NamedMask: Distilling Segmenters from Complementary Foundation Modelscited by 20
  27. 2022MOVE: Unsupervised Movable Object Segmentation and Detectioncited by 20
  28. 2024Towards Open-Vocabulary Semantic Segmentation Without Semantic Labelscited by 19
  29. 2023Causal Unsupervised Semantic Segmentationcited by 18
  30. 2023SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentationcited by 18
  31. 2023MOST: Multiple Object localization with Self-supervised Transformers for object discoverycited by 17
  32. 2024DIVE: Taming DINO for Subject-Driven Video Editingcited by 17
  33. 2023Delving into Shape-aware Zero-shot Semantic Segmentationcited by 16
  34. 2023Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Surveycited by 16
  35. 2022A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentationcited by 15
  36. 2022Far Away in the Deep Space: Dense Nearest-Neighbor-Based Out-of-Distribution Detectioncited by 15
  37. 2024Near, far: Patch-ordering enhances vision foundation models' scene understandingcited by 15
  38. 2025Exploring Temporally-Aware Features for Point Trackingcited by 15

50 most cited of 150 All 150 (JSON)