# DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut

arXiv: https://arxiv.org/abs/2406.02842 · HTML: https://mlsanity.com/paper/2406.02842/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 83 of its 93 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 30 of 83)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)

## Built on by (most cited first, 26 of 26)

- [Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers](https://mlsanity.com/paper/2509.18096/) (2025)
- [JAFAR: Jack up Any Feature at Any Resolution](https://mlsanity.com/paper/2506.11136/) (2025)
- [GLASS: Guided Latent Slot Diffusion for Object-Centric Learning](https://mlsanity.com/paper/2407.17929/) (2024)
- [Can Generative Geospatial Diffusion Models Excel as Discriminative Geospatial Foundation Models?](https://mlsanity.com/paper/2503.07890/) (2025)
- [Beyond Single Images: Retrieval Self-Augmented Unsupervised Camouflaged Object Detection](https://mlsanity.com/paper/2510.18437/) (2025)
- [TRACE: Your Diffusion Model is Secretly an Instance Edge Detector](https://mlsanity.com/paper/2503.07982/) (2025)
- [Diffusion Model as a Generalist Segmentation Learner](https://mlsanity.com/paper/2604.24575/) (2026)
- [Unsupervised Segmentation by Diffusing, Walking and Cutting](https://mlsanity.com/paper/2412.04678/) (2024)
- [DIP: Unsupervised Dense In-Context Post-training of Visual Representations](https://mlsanity.com/paper/2506.18463/) (2025)
- [Making Training-Free Diffusion Segmentors Scale with the Generative Power](https://mlsanity.com/paper/2603.06178/) (2026)
- [Latent Diffusion U-Net Representations Contain Positional Embeddings and Anomalies](https://mlsanity.com/paper/2504.07008/) (2025)
- [FA-Seg: A Fast and Accurate Diffusion-Based Method for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2506.23323/) (2025)
- [Temporally Consistent Unsupervised Segmentation for Mobile Robot Perception](https://mlsanity.com/paper/2507.22194/) (2025)
- [Scene-Centric Unsupervised Video Panoptic Segmentation](https://mlsanity.com/paper/2606.04925/) (2026)
- [Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentation](https://mlsanity.com/paper/2606.15590/) (2026)
- [Falcon: Fractional Alternating Cut with Overcoming Minima in Unsupervised Segmentation](https://mlsanity.com/paper/2504.05613/) (2025)
- [Enhancing Object Discovery for Unsupervised Instance Segmentation and Object Detection](https://mlsanity.com/paper/2508.02386/) (2025)
- [IELDG: Suppressing Domain-Specific Noise with Inverse Evolution Layers for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2508.19604/) (2025)
- [A Training-Free Framework for Open-Vocabulary Image Segmentation and Recognition with EfficientNet and CLIP](https://mlsanity.com/paper/2510.19333/) (2025)
- [Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos](https://mlsanity.com/paper/2511.19936/) (2025)
- [SPARK: Stochastic Propagation via Affinity-guided Random walK for training-free unsupervised segmentation](https://mlsanity.com/paper/2602.00516/) (2026)
- [Excite, Attend and Segment (EASe): Domain-Agnostic Fine-Grained Mask Discovery with Feature Calibration and Self-Supervised Upsampling](https://mlsanity.com/paper/2604.00276/) (2026)
- [One Click per Cell Type Suffices: Training-free Group Interaction for Cell Instance Segmentation](https://mlsanity.com/paper/2605.29429/) (2026)
- [High-Fidelity Synthetic Transmission Electron Microscopy Image Generation Using Diffusion Probabilistic Models for Data-Limited Semiconductor Metrology](https://mlsanity.com/paper/2606.24817/) (2026)
- [PixelUp: Zero-Shot Semantic Feature Upsampling for Fine-Grained Vision Tasks](https://mlsanity.com/paper/2608.02792/) (2026)
- [Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision](https://mlsanity.com/paper/2609.06491/) (2026)

## Abstract (arXiv)

Foundation models have emerged as powerful tools across various domains including language, vision, and multimodal tasks. While prior works have addressed unsupervised image segmentation, they significantly lag behind supervised models. In this paper, we use a diffusion UNet encoder as a foundation vision encoder and introduce DiffCut, an unsupervised zero-shot segmentation method that solely harnesses the output features from the final self-attention block. Through extensive experimentation, we demonstrate that the utilization of these diffusion features in a graph based segmentation algorithm, significantly outperforms previous state-of-the-art methods on zero-shot segmentation. Specifically, we leverage a recursive Normalized Cut algorithm that softly regulates the granularity of detected objects and produces well-defined segmentation maps that precisely capture intricate image details. Our work highlights the remarkably accurate semantic knowledge embedded within diffusion UNet encoders that could then serve as foundation vision encoders for downstream tasks. Project page at https://diffcut-segmentation.github.io