mlsanityData to 7 Oct 2026

Repurposing SAM for User-Defined Semantics Aware Segmentation

arXiv 2312.02420 ↗First posted 5 Dec 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

The Segment Anything Model (SAM) excels at generating precise object masks from input prompts but lacks semantic awareness, failing to associate its generated masks with specific object categories. To address this limitation, we propose U-SAM, a novel framework that imbibes semantic awareness into SAM, enabling it to generate targeted masks for user-specified object categories. Given only object class names as input from the user, U-SAM provides pixel-level semantic annotations for images without requiring any labeled/unlabeled samples from the test data distribution. Our approach leverages synthetically generated or web crawled images to accumulate semantic information about the desired object classes. We then learn a mapping function between SAM's mask embeddings and object class labels, effectively enhancing SAM with granularity-specific semantic recognition capabilities. As a result, users can obtain meaningful and targeted segmentation masks for specific objects they request, rather than generic and unlabeled masks. We evaluate U-SAM on PASCAL VOC 2012 and MSCOCO-80, achieving significant mIoU improvements of +17.95% and +5.20%, respectively, over state-of-the-art methods. By transforming SAM into a semantically aware segmentation model, U-SAM offers a practical and flexible solution for pixel-level annotation across diverse and unseen domains in a resource-constrained environment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

43 of 59 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023Segment Anythingcited by 8,158
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  11. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  12. 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
31 more
  1. 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
  2. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  3. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163
  4. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  5. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  6. 2022TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cutcited by 111
  7. 2019Large-scale interactive object segmentation with human annotatorscited by 109
  8. 2021Large-scale Unsupervised Semantic Segmentationcited by 98
  9. 2019Fast Interactive Object Annotation with Curve-GCNcited by 95
  10. 2023Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Groupingcited by 92
  11. 2022Self-Supervised Learning of Object Parts for Semantic Segmentationcited by 90
  12. 2022CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentationcited by 60
  13. 2023Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Reviewcited by 57
  14. 2022Unsupervised Semantic Segmentation with Self-supervised Object-centric Representationscited by 54
  15. 2023Text2Seg: Remote Sensing Image Semantic Segmentation via Text-Guided Visual Foundation Modelscited by 44
  16. 2021MetaCorrection: Domain-aware Meta Loss Correction for Unsupervised Domain Adaptation in Semantic Segmentationcited by 43
  17. 2022ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentationcited by 42
  18. 2023Leveraging Hidden Positives for Unsupervised Semantic Segmentationcited by 40
  19. 2022Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformerscited by 39
  20. 2021TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentationcited by 38
  21. 2023CLIP-S4: Language-Guided Self-Supervised Semantic Segmentationcited by 34
  22. 2022DeepCut: Unsupervised Segmentation using Graph Neural Networks Clusteringcited by 25
  23. 2020Unsupervised Domain Adaptation for Semantic Segmentation by Content Transfercited by 24
  24. 2022Drive&Segment: Unsupervised Semantic Segmentation of Urban Scenes via Cross-modal Distillationcited by 19
  25. 2021Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasetscited by 17
  26. 2023Unsupervised Domain Adaptation for Semantic Segmentation with Pseudo Label Self-Refinementcited by 16
  27. 2022MFSNet: A Multi Focus Segmentation Network for Skin Lesion Segmentationcited by 10
  28. 2022Rethinking Alignment and Uniformity in Unsupervised Semantic Segmentationcited by 10
  29. 2022IDEAL: Improved DEnse locAL Contrastive Learning for Semi-Supervised Medical Image Segmentationcited by 5
  30. 2023Recognize Any Regionscited by 5
  31. 2023POISE: Pose Guided Human Silhouette Extraction under Occlusionscited by 5

Built on by

Not cited yet.