# An Alternative to WSSS? An Empirical Study of the Segment Anything Model (SAM) on Weakly-Supervised Semantic Segmentation Problems

arXiv: https://arxiv.org/abs/2305.01586 · HTML: https://mlsanity.com/paper/2305.01586/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 54 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1803.10464/) (2018)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations](https://mlsanity.com/paper/1904.05044/) (2019)
- [BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1503.01640/) (2015)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)
- [Simple Does It: Weakly Supervised Instance and Semantic Segmentation](https://mlsanity.com/paper/1603.07485/) (2016)
- [Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach](https://mlsanity.com/paper/1703.08448/) (2017)
- [Multi-class Token Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02891/) (2022)
- [Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2103.08896/) (2021)
- [Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2105.08965/) (2021)
- [On Regularized Losses for Weakly-supervised CNN Segmentation](https://mlsanity.com/paper/1803.09569/) (2018)
- [Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.02664/) (2022)
- [CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2212.09506/) (2022)
- [BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation](https://mlsanity.com/paper/2103.08907/) (2021)
- [Learning random-walk label propagation for weakly-supervised semantic segmentation](https://mlsanity.com/paper/1802.00470/) (2018)
- [Box-driven Class-wise Region Masking and Filling Rate Guided Loss for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1904.11693/) (2019)
- [Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2103.14581/) (2021)
- [Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02909/) (2022)
- [Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast](https://mlsanity.com/paper/2110.07110/) (2021)
- [Weakly-Supervised Semantic Segmentation by Iteratively Mining Common Object Features](https://mlsanity.com/paper/1806.04659/) (2018)
- [Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2107.11787/) (2021)
- [Complementary Patch for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2108.03852/) (2021)
- [Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.00962/) (2022)

## Built on by (most cited first, 19 of 19)

- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering](https://mlsanity.com/paper/2306.06211/) (2023)
- [Foundation Model Assisted Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2312.03585/) (2023)
- [WeakSAM: Segment Anything Meets Weakly-supervised Instance-level Recognition](https://mlsanity.com/paper/2402.14812/) (2024)
- [Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors](https://mlsanity.com/paper/2505.24103/) (2025)
- [Tuning-free Universally-Supervised Semantic Segmentation](https://mlsanity.com/paper/2405.14294/) (2024)
- [Detecting Wildfires on UAVs with Real-time Segmentation Trained by Larger Teacher Models](https://mlsanity.com/paper/2408.10843/) (2024)
- [Industrial Segment Anything -- a Case Study in Aircraft Manufacturing, Intralogistics, Maintenance, Repair, and Overhaul](https://mlsanity.com/paper/2307.12674/) (2023)
- [Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2309.01369/) (2023)
- [WPS-SAM: Towards Weakly-Supervised Part Segmentation with Foundation Models](https://mlsanity.com/paper/2407.10131/) (2024)
- [All-pairs Consistency Learning for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2308.04321/) (2023)
- [Semantic Prompt Learning for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2401.11791/) (2024)
- [DHR: Dual Features-Driven Hierarchical Rebalancing in Inter- and Intra-Class Regions for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2404.00380/) (2024)
- [Boosting Weakly-Supervised Image Segmentation via Representation, Transform, and Compensator](https://mlsanity.com/paper/2309.00871/) (2023)
- [Segment Anything Model-guided Collaborative Learning Network for Scribble-supervised Polyp Segmentation](https://mlsanity.com/paper/2312.00312/) (2023)
- [Weakly Supervised Segmentation as Semantic-Based Regularization](https://mlsanity.com/paper/2605.13674/) (2026)
- [Top-GAP: Integrating Size Priors in CNNs for more Interpretability, Robustness, and Bias Mitigation](https://mlsanity.com/paper/2409.04819/) (2024)
- [Depth Edge Alignment Loss: DEALing with Depth in Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2509.17702/) (2025)
- [CRF Loss is How Networks Should Learn Boundaries in Weakly Supervised Segmentation](https://mlsanity.com/paper/2609.34183/) (2026)

## Abstract (arXiv)

The Segment Anything Model (SAM) has demonstrated exceptional performance and versatility, making it a promising tool for various related tasks. In this report, we explore the application of SAM in Weakly-Supervised Semantic Segmentation (WSSS). Particularly, we adapt SAM as the pseudo-label generation pipeline given only the image-level class labels. While we observed impressive results in most cases, we also identify certain limitations. Our study includes performance evaluations on PASCAL VOC and MS-COCO, where we achieved remarkable improvements over the latest state-of-the-art methods on both datasets. We anticipate that this report encourages further explorations of adopting SAM in WSSS, as well as wider real-world applications.