# Zero-Shot Edge Detection with SCESAME: Spectral Clustering-based Ensemble for Segment Anything Model Estimation

arXiv: https://arxiv.org/abs/2308.13779 · HTML: https://mlsanity.com/paper/2308.13779/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 64 references that are in the map.
- First posted on arXiv 26 Aug 2023.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [Pixel Difference Networks for Efficient Edge Detection](https://mlsanity.com/paper/2108.07009/) (2021)
- [Richer Convolutional Features for Edge Detection](https://mlsanity.com/paper/1612.02103/) (2016)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [Learning to predict crisp boundaries](https://mlsanity.com/paper/1807.10097/) (2018)
- [Boundary-preserving Mask R-CNN](https://mlsanity.com/paper/2007.08921/) (2020)
- [Bi-Directional Cascade Network for Perceptual Edge Detection](https://mlsanity.com/paper/1902.10903/) (2019)
- [Fast Edge Detection Using Structured Forests](https://mlsanity.com/paper/1406.5549/) (2014)
- [Segment Any Anomaly without Training via Hybrid Prompt Regularization](https://mlsanity.com/paper/2305.10724/) (2023)
- [DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection](https://mlsanity.com/paper/1412.1123/) (2014)
- [EDTER: Edge Detection with Transformer](https://mlsanity.com/paper/2203.08566/) (2022)
- [Object Contour Detection with a Fully Convolutional Encoder-Decoder Network](https://mlsanity.com/paper/1603.04530/) (2016)
- [Dynamic Feature Integration for Simultaneous Detection of Salient Object, Edge and Skeleton](https://mlsanity.com/paper/2004.08595/) (2020)
- [Learning Deep Structured Multi-Scale Features using Attention-Gated CRFs for Contour Prediction](https://mlsanity.com/paper/1801.00524/) (2018)
- [Oriented Edge Forests for Boundary Detection](https://mlsanity.com/paper/1412.4181/) (2014)
- [The Treasure Beneath Multiple Annotations: An Uncertainty-aware Edge Detector](https://mlsanity.com/paper/2303.11828/) (2023)
- [High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision](https://mlsanity.com/paper/1504.06201/) (2015)
- [SemiContour: A Semi-supervised Learning Approach for Contour Detection](https://mlsanity.com/paper/1605.04996/) (2016)
- [Pixel-wise Deep Learning for Contour Detection](https://mlsanity.com/paper/1504.01989/) (2015)

## Built on by (most cited first, 2 of 2)

- [Subobject-level Image Tokenization](https://mlsanity.com/paper/2402.14327/) (2024)
- [Generative Edge Detection with Stable Diffusion](https://mlsanity.com/paper/2410.03080/) (2024)

## Abstract (arXiv)

This paper proposes a novel zero-shot edge detection with SCESAME, which stands for Spectral Clustering-based Ensemble for Segment Anything Model Estimation, based on the recently proposed Segment Anything Model (SAM). SAM is a foundation model for segmentation tasks, and one of the interesting applications of SAM is Automatic Mask Generation (AMG), which generates zero-shot segmentation masks of an entire image. AMG can be applied to edge detection, but suffers from the problem of overdetecting edges. Edge detection with SCESAME overcomes this problem by three steps: (1) eliminating small generated masks, (2) combining masks by spectral clustering, taking into account mask positions and overlaps, and (3) removing artifacts after edge detection. We performed edge detection experiments on two datasets, BSDS500 and NYUDv2. Although our zero-shot approach is simple, the experimental results on BSDS500 showed almost identical performance to human performance and CNN-based methods from seven years ago. In the NYUDv2 experiments, it performed almost as well as recent CNN-based methods. These results indicate that our method effectively enhances the utility of SAM and can be a new direction in zero-shot edge detection methods.