# Segment Anything without Supervision

arXiv: https://arxiv.org/abs/2406.20081 · HTML: https://mlsanity.com/paper/2406.20081/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Sequential Modeling Enables Scalable Learning for Large Vision Models](https://mlsanity.com/paper/2312.00785/) (2023)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [Cut and Learn for Unsupervised Object Detection and Instance Segmentation](https://mlsanity.com/paper/2301.11320/) (2023)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [PACO: Parts and Attributes of Common Objects](https://mlsanity.com/paper/2301.01795/) (2023)
- [CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement](https://mlsanity.com/paper/2005.02551/) (2020)
- [TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut](https://mlsanity.com/paper/2209.00383/) (2022)
- [PartImageNet: A Large, High-Quality Dataset of Parts](https://mlsanity.com/paper/2112.00933/) (2021)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [Open-World Entity Segmentation](https://mlsanity.com/paper/2107.14228/) (2021)
- [Unsupervised Object Discovery and Localization in the Wild: Part-based Matching with Bottom-up Region Proposals](https://mlsanity.com/paper/1501.06170/) (2015)
- [Toward unsupervised, multi-object discovery in large-scale image collections](https://mlsanity.com/paper/2007.02662/) (2020)
- [Discovering Object Masks with Transformers for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2206.06363/) (2022)

## Built on by (most cited first, 28 of 28)

- [Scaling Vision Pre-Training to 4K Resolution](https://mlsanity.com/paper/2503.19903/) (2025)
- [UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation](https://mlsanity.com/paper/2506.09284/) (2025)
- [Scene-Centric Unsupervised Panoptic Segmentation](https://mlsanity.com/paper/2504.01955/) (2025)
- [Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation](https://mlsanity.com/paper/2411.10411/) (2024)
- [CutS3D: Cutting Semantics in 3D for 2D Unsupervised Instance Segmentation](https://mlsanity.com/paper/2411.16319/) (2024)
- [Solving Instance Detection from an Open-World Perspective](https://mlsanity.com/paper/2503.00359/) (2025)
- [TRACE: Your Diffusion Model is Secretly an Instance Edge Detector](https://mlsanity.com/paper/2503.07982/) (2025)
- [unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning](https://mlsanity.com/paper/2506.01778/) (2025)
- [CycleSAM: Few-Shot Surgical Scene Segmentation with Cycle- and Scene-Consistent Feature Matching](https://mlsanity.com/paper/2407.06795/) (2024)
- [E-SAM: Training-Free Segment Every Entity Model](https://mlsanity.com/paper/2503.12094/) (2025)
- [FlowCut: Unsupervised Video Instance Segmentation via Temporal Mask Matching](https://mlsanity.com/paper/2505.13174/) (2025)
- [gen2seg: Generative Models Enable Generalizable Instance Segmentation](https://mlsanity.com/paper/2505.15263/) (2025)
- [M2N2V2: Multi-Modal Unsupervised and Training-free Interactive Segmentation](https://mlsanity.com/paper/2503.16254/) (2025)
- [SynthFM: Training Modality-agnostic Foundation Models for Medical Image Segmentation without Real Medical Data](https://mlsanity.com/paper/2504.08177/) (2025)
- [Open-set Anomaly Segmentation in Complex Scenarios](https://mlsanity.com/paper/2504.19706/) (2025)
- [Hierarchical Compact Clustering Attention (COCA) for Unsupervised Object-Centric Learning](https://mlsanity.com/paper/2505.02071/) (2025)
- [SMOL-MapSeg: Show Me One Label as prompt](https://mlsanity.com/paper/2508.05501/) (2025)
- [Scene-Centric Unsupervised Video Panoptic Segmentation](https://mlsanity.com/paper/2606.04925/) (2026)
- [Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentation](https://mlsanity.com/paper/2606.15590/) (2026)
- [AutoExpert: Automating 3D LiDAR Annotation from Expert-Crafted Guidelines](https://mlsanity.com/paper/2506.02914/) (2025)
- [CLASP: Adaptive Spectral Clustering for Unsupervised Per-Image Segmentation](https://mlsanity.com/paper/2509.25016/) (2025)
- [Pancakes: Consistent Multi-Protocol Image Segmentation Across Biomedical Domains](https://mlsanity.com/paper/2512.13534/) (2025)
- [High-Resolution Image Reconstruction with Unsupervised Learning and Noisy Data Applied to Ion-Beam Dynamics for Particle Accelerators](https://mlsanity.com/paper/2603.06689/) (2026)
- [Dissecting Model Failures in Abdominal Aortic Aneurysm Segmentation through Explainability-Driven Analysis](https://mlsanity.com/paper/2603.24801/) (2026)
- [Excite, Attend and Segment (EASe): Domain-Agnostic Fine-Grained Mask Discovery with Feature Calibration and Self-Supervised Upsampling](https://mlsanity.com/paper/2604.00276/) (2026)
- [Can Frozen Hyperspherical Features Guide the Selection of Pseudo Masks?](https://mlsanity.com/paper/2609.30080/) (2026)
- [Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision](https://mlsanity.com/paper/2609.39785/) (2026)
- [TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision](https://mlsanity.com/paper/2610.14651/) (2026)

## Abstract (arXiv)

The Segmentation Anything Model (SAM) requires labor-intensive data labeling. We present Unsupervised SAM (UnSAM) for promptable and automatic whole-image segmentation that does not require human annotations. UnSAM utilizes a divide-and-conquer strategy to "discover" the hierarchical structure of visual scenes. We first leverage top-down clustering methods to partition an unlabeled image into instance/semantic level segments. For all pixels within a segment, a bottom-up clustering method is employed to iteratively merge them into larger groups, thereby forming a hierarchical structure. These unsupervised multi-granular masks are then utilized to supervise model training. Evaluated across seven popular datasets, UnSAM achieves competitive results with the supervised counterpart SAM, and surpasses the previous state-of-the-art in unsupervised segmentation by 11% in terms of AR. Moreover, we show that supervised SAM can also benefit from our self-supervised labels. By integrating our unsupervised pseudo masks into SA-1B's ground-truth masks and training UnSAM with only 1% of SA-1B, a lightly semi-supervised UnSAM can often segment entities overlooked by supervised SAM, exceeding SAM's AR by over 6.7% and AP by 3.9% on SA-1B.