SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches
Not on the map. Why?
Abstract (arXiv)
We introduce SketchYourSeg, a novel framework that establishes freehand sketches as a powerful query modality for subjective image segmentation across entire galleries through a single exemplar sketch. Unlike text prompts that struggle with spatial specificity or interactive methods confined to single-image operations, sketches naturally combine semantic intent with structural precision. This unique dual encoding enables precise visual disambiguation for segmentation tasks where text descriptions would be cumbersome or ambiguous -- such as distinguishing between visually similar instances, specifying exact part boundaries, or indicating spatial relationships in composed concepts. Our approach addresses three fundamental challenges: (i) eliminating the need for pixel-perfect annotation masks during training with a mask-free framework; (ii) creating a synergistic relationship between sketch-based image retrieval (SBIR) models and foundation models (CLIP/DINOv2) where the former provides training signals while the latter generates masks; and (iii) enabling multi-granular segmentation capabilities through purpose-made sketch augmentation strategies. Our extensive evaluations demonstrate superior performance over existing approaches across diverse benchmarks, establishing a new paradigm for user-guided image segmentation that balances precision with efficiency.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 74 of its 93 references that are in the map. First posted on arXiv 27 Jan 2025.
Builds on
74 of 93 references.
References · 74
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2017Mask R-CNNcited by 7,259
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
- 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
38 more
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
- 2015Learning Deep Features for Discriminative Localizationcited by 2,430
- 2021Learning to Prompt for Vision-Language Modelscited by 2,269
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
- 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
- 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
- 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
- 2023Segment Anything in High Qualitycited by 373
- 2016ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentationcited by 330
- 2017A Neural Representation of Sketch Drawingscited by 311
- 2015What's the Point: Semantic Segmentation with Point Supervisioncited by 297
- 2023Personalize Segment Anything Model with One Shotcited by 250
- 2015BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentationcited by 238
- 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
- 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
- 2015Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentationcited by 189
- 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
- 2021Localizing Visual Sounds the Hard Waycited by 169
- 2018Tell Me Where to Look: Guided Attention Inference Networkcited by 168
- 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
- 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
- 2018Learning random-walk label propagation for weakly-supervised semantic segmentationcited by 101
- 2015Constrained Convolutional Neural Networks for Weakly Supervised Segmentationcited by 100
- 2019SCOPS: Self-Supervised Co-Part Segmentationcited by 100
- 2019SegSort: Segmentation by Discriminative Sorting of Segmentscited by 96
- 2019Doodle to Search: Practical Zero-Shot Sketch-based Image Retrievalcited by 79
- 2023CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Notcited by 79
- 2019Unsupervised Object Segmentation by Redrawingcited by 74
- 2023SKED: Sketch-guided Text-based 3D Editingcited by 72
- 2018A Zero-Shot Framework for Sketch-based Image Retrievalcited by 70
- 2017Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrievalcited by 68
- 2020Self-Supervised Visual Representation Learning from Hierarchical Groupingcited by 63
- 2020Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrievalcited by 59
- 2021Labels4Free: Unsupervised Segmentation using StyleGANcited by 53
- 2020Object Segmentation Without Labels with Large-Scale Generative Modelscited by 52
- 2021StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrievalcited by 50
- 2021Recall@k Surrogate Loss with Large Batches and Similarity Mixupcited by 49
- 2022FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Contextcited by 47
50 most cited of 74 All 74 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.