mlsanityData to 14 Oct 2026

Tuning-free Universally-Supervised Semantic Segmentation

arXiv 2405.14294 ↗First posted 23 May 2024cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

This work presents a tuning-free semantic segmentation framework based on classifying SAM masks by CLIP, which is universally applicable to various types of supervision. Initially, we utilize CLIP's zero-shot classification ability to generate pseudo-labels or perform open-vocabulary segmentation. However, the misalignment between mask and CLIP text embeddings leads to suboptimal results. To address this issue, we propose discrimination-bias aligned CLIP to closely align mask and text embedding, offering an overhead-free performance gain. We then construct a global-local consistent classifier to classify SAM masks, which reveals the intrinsic structure of high-quality embeddings produced by DBA-CLIP and demonstrates robustness against noisy pseudo-labels. Extensive experiments validate the efficiency and effectiveness of our method, and we achieve state-of-the-art (SOTA) or competitive performance across various datasets and supervision types.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 77 of its 95 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

77 of 95 references.

References · 77

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2023Segment Anythingcited by 8,234
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  12. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
38 more
  1. 2017Billion-scale similarity search with GPUscited by 2,497
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  3. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  4. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  5. 2021Extract Free Dense Labels from CLIPcited by 583
  6. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  7. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  8. 2024The Faiss librarycited by 376
  9. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 352
  10. 2018Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentationcited by 321
  11. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  12. 2023Fast Segment Anythingcited by 290
  13. 2019Label Propagation for Deep Semi-supervised Learningcited by 289
  14. 2020PseudoSeg: Designing Pseudo Labels for Semantic Segmentationcited by 180
  15. 2022Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentationcited by 178
  16. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
  17. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  18. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  19. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  20. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
  21. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  22. 2022Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformerscited by 118
  23. 2022CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentationcited by 118
  24. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  25. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  26. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  27. 2021Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentationcited by 101
  28. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 96
  29. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  30. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 91
  31. 2019Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approachcited by 88
  32. 2022Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentationcited by 87
  33. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 75
  34. 2022Cross Language Image Matching for Weakly Supervised Semantic Segmentationcited by 72
  35. 2023Token Contrast for Weakly-Supervised Semantic Segmentationcited by 65
  36. 2022Augmentation Matters: A Simple-yet-Effective Approach to Semi-supervised Semantic Segmentationcited by 64
  37. 2023Learning Mask-aware CLIP Representations for Zero-Shot Segmentationcited by 57
  38. 2023Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentationcited by 53

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2024ResCLIP: Residual Attention for Training-free Dense Vision-language Inferencecited by 29
  2. 2025Exploring the Underwater World Segmentation without Extra Trainingcited by 16
  3. 2025Hi-LSplat: Hierarchical 3D Language Gaussian Splattingcited by 4
  4. 2026Enabling Training-Free Text-Based Remote Sensing Segmentationcited by 2
  5. 2025A Survey on Training-free Open-Vocabulary Semantic Segmentationcited by 0
  6. 2026Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Modelscited by 0