mlsanityData to 14 Oct 2026

CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation

arXiv 2212.09506 ↗First posted 16 Dec 2022cs.CV cs.AIcited by 118

Not on the map. Why?

Abstract (arXiv)

Weakly supervised semantic segmentation (WSSS) with image-level labels is a challenging task. Mainstream approaches follow a multi-stage framework and suffer from high training costs. In this paper, we explore the potential of Contrastive Language-Image Pre-training models (CLIP) to localize different categories with only image-level labels and without further training. To efficiently generate high-quality segmentation masks from CLIP, we propose a novel WSSS framework called CLIP-ES. Our framework improves all three stages of WSSS with special designs for CLIP: 1) We introduce the softmax function into GradCAM and exploit the zero-shot ability of CLIP to suppress the confusion caused by non-target classes and backgrounds. Meanwhile, to take full advantage of CLIP, we re-explore text inputs under the WSSS setting and customize two text-driven strategies: sharpness-based prompt selection and synonym fusion. 2) To simplify the stage of CAM refinement, we propose a real-time class-aware attention-based affinity (CAA) module based on the inherent multi-head self-attention (MHSA) in CLIP-ViTs. 3) When training the final segmentation model with the masks generated by CLIP, we introduced a confidence-guided loss (CGL) focus on confident regions. Our CLIP-ES achieves SOTA performance on Pascal VOC 2012 and MS COCO 2014 while only taking 10% time of previous methods for the pseudo mask generation. Code is available at https://github.com/linyq2117/CLIP-ES.

Key facts. Cited by 118 papers in the map (data through 14 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 16 Dec 2022.

Builds on

42 of 54 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  7. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  8. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  9. 2016ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentationcited by 331
  10. 2018Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentationcited by 321
  11. 2020Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentationcited by 300
  12. 2015What's the Point: Semantic Segmentation with Point Supervisioncited by 298
30 more
  1. 2019Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relationscited by 247
  2. 2015BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentationcited by 238
  3. 2017Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approachcited by 201
  4. 2015Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentationcited by 189
  5. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  6. 2020Weakly-Supervised Semantic Segmentation via Sub-category Explorationcited by 145
  7. 2020Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentationcited by 144
  8. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  9. 2021Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentationcited by 134
  10. 2021Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentationcited by 127
  11. 2022Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformerscited by 118
  12. 2018Learning random-walk label propagation for weakly-supervised semantic segmentationcited by 101
  13. 2018Self-Erasing Network for Integral Object Attentioncited by 95
  14. 2021Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentationcited by 95
  15. 2019Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approachcited by 88
  16. 2022Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentationcited by 87
  17. 2021Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrastcited by 85
  18. 2018CIAN: Cross-Image Affinity Net for Weakly Supervised Semantic Segmentationcited by 80
  19. 2021Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentationcited by 77
  20. 2022Class Re-Activation Maps for Weakly-Supervised Semantic Segmentationcited by 75
  21. 2022Cross Language Image Matching for Weakly Supervised Semantic Segmentationcited by 72
  22. 2021Reducing Information Bottleneck for Weakly Supervised Semantic Segmentationcited by 65
  23. 2022Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentationcited by 64
  24. 2022Weakly Supervised Semantic Segmentation using Out-of-Distribution Datacited by 58
  25. 2022L2G: A Simple Local-to-Global Knowledge Transfer Framework for Weakly Supervised Semantic Segmentationcited by 57
  26. 2020Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentationcited by 52
  27. 2021Discriminative Region Suppression for Weakly-Supervised Semantic Segmentationcited by 50
  28. 2021Pseudo-mask Matters in Weakly-supervised Semantic Segmentationcited by 48
  29. 2022Threshold Matters in WSSS: Manipulating the Activation for the Robust and Accurate Segmentation Model Against Thresholdscited by 37
  30. 2021Uncertainty Estimation via Response Scaling for Pseudo-mask Noise Mitigation in Weakly-supervised Semantic Segmentationcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 25; 2024: 42; 2025: 37; 2026: 14.

118 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 91
  3. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  4. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 73
  5. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  6. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  7. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  8. 2023Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentationcited by 53
  9. 2024Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detectioncited by 49
  10. 2023CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentationcited by 44
  11. 2023SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidancecited by 41
  12. 2024MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentationcited by 39
38 more
  1. 2023Referring Image Segmentation Using Text Supervisioncited by 35
  2. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  3. 2023Foundation Model Assisted Weakly Supervised Semantic Segmentationcited by 33
  4. 2024PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaborationcited by 28
  5. 2025ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classificationcited by 28
  6. 2023Segment Anything is A Good Pseudo-label Generator for Weakly Supervised Semantic Segmentationcited by 27
  7. 2023TextFusion: Unveiling the Power of Textual Semantics for Controllable Image Fusioncited by 26
  8. 2023NLLB-CLIP -- train performant multilingual image retrieval model on a budgetcited by 24
  9. 2023WeakTr: Exploring Plain Vision Transformer for Weakly-supervised Semantic Segmentationcited by 22
  10. 2023MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentationcited by 22
  11. 2024Textual Query-Driven Mask Transformer for Domain Generalized Segmentationcited by 21
  12. 2024Rethinking Prior Information Generation with CLIP for Few-Shot Segmentationcited by 20
  13. 2025SAMRefiner: Taming Segment Anything Model for Universal Mask Refinementcited by 20
  14. 2023An Alternative to WSSS? An Empirical Study of the Segment Anything Model (SAM) on Weakly-Supervised Semantic Segmentation Problemscited by 19
  15. 2023Weakly-Supervised Semantic Segmentation with Image-Level Labels: from Traditional Models to Foundation Modelscited by 19
  16. 2023Few-Shot Panoptic Segmentation With Foundation Modelscited by 18
  17. 2024Spatial Structure Constraints for Weakly Supervised Semantic Segmentationcited by 17
  18. 2024DuPL: Dual Student with Trustworthy Progressive Learning for Robust Weakly Supervised Semantic Segmentationcited by 17
  19. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentationcited by 17
  20. 2023From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Modelscited by 16
  21. 2024Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentationcited by 16
  22. 2024Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentationcited by 15
  23. 2024DetailCLIP: Detail-Oriented CLIP for Fine-Grained Taskscited by 15
  24. 2024Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentationcited by 14
  25. 2024A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIPcited by 14
  26. 2025Refining CLIP's Spatial Awareness: A Visual-Centric Perspectivecited by 13
  27. 2024WeakSAM: Segment Anything Meets Weakly-supervised Instance-level Recognitioncited by 10
  28. 2024iSeg: An Iterative Refinement-based Framework for Training-free Segmentationcited by 10
  29. 2024Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehensioncited by 10
  30. 2023CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentationcited by 9
  31. 20252HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videoscited by 9
  32. 2025Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priorscited by 9
  33. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentationcited by 9
  34. 2023DiPS: Discriminative Pseudo-Label Sampling with Self-Supervised Transformers for Weakly Supervised Object Localizationcited by 8
  35. 2024Knowledge Transfer with Simulated Inter-Image Erasing for Weakly Supervised Semantic Segmentationcited by 8
  36. 2025Exploring CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentationcited by 8
  37. 2023CoLo-CAM: Class Activation Mapping for Object Co-Localization in Weakly-Labeled Unconstrained Videoscited by 7
  38. 2023Label-efficient Segmentation via Affinity Propagationcited by 7

50 most cited of 118 All 118 (JSON)