# Token Contrast for Weakly-Supervised Semantic Segmentation

arXiv: https://arxiv.org/abs/2303.01267 · HTML: https://mlsanity.com/paper/2303.01267/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 54 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1803.10464/) (2018)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations](https://mlsanity.com/paper/1904.05044/) (2019)
- [Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach](https://mlsanity.com/paper/1703.08448/) (2017)
- [Single-Stage Semantic Segmentation from Image Labels](https://mlsanity.com/paper/2005.08104/) (2020)
- [Weakly-Supervised Semantic Segmentation via Sub-category Exploration](https://mlsanity.com/paper/2008.01183/) (2020)
- [Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2007.01947/) (2020)
- [Multi-class Token Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02891/) (2022)
- [Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice](https://mlsanity.com/paper/2203.05962/) (2022)
- [Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2105.08965/) (2021)
- [TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization](https://mlsanity.com/paper/2103.14862/) (2021)
- [On Regularized Losses for Weakly-supervised CNN Segmentation](https://mlsanity.com/paper/1803.09569/) (2018)
- [Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.02664/) (2022)
- [BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation](https://mlsanity.com/paper/2103.08907/) (2021)
- [Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2103.14581/) (2021)
- [Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach](https://mlsanity.com/paper/1911.08039/) (2019)

## Built on by (most cited first, 30 of 65)

- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [The Hidden Attention of Mamba Models](https://mlsanity.com/paper/2403.01590/) (2024)
- [Foundation Model Assisted Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2312.03585/) (2023)
- [Measure-to-measure interpolation using Transformers](https://mlsanity.com/paper/2411.04551/) (2024)
- [WeakTr: Exploring Plain Vision Transformer for Weakly-supervised Semantic Segmentation](https://mlsanity.com/paper/2304.01184/) (2023)
- [MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2308.03005/) (2023)
- [Weakly-Supervised Semantic Segmentation with Image-Level Labels: from Traditional Models to Foundation Models](https://mlsanity.com/paper/2310.13026/) (2023)
- [Spatial Structure Constraints for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2401.11122/) (2024)
- [Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2403.07630/) (2024)
- [Prompt Categories Cluster for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2412.13823/) (2024)
- [Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2406.11189/) (2024)
- [Adaptive Patch Contrast for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2407.10649/) (2024)
- [Image Augmentation Agent for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2412.20439/) (2024)
- [GLASS: Guided Latent Slot Diffusion for Object-Centric Learning](https://mlsanity.com/paper/2407.17929/) (2024)
- [Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation](https://mlsanity.com/paper/2405.16504/) (2024)
- [Top-K Pooling with Patch Contrastive Learning for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2310.09828/) (2023)
- [iSeg: An Iterative Refinement-based Framework for Training-free Segmentation](https://mlsanity.com/paper/2409.03209/) (2024)
- [Boosting Unsupervised Semantic Segmentation with Principal Mask Proposals](https://mlsanity.com/paper/2404.16818/) (2024)
- [Using Unreliable Pseudo-Labels for Label-Efficient Semantic Segmentation](https://mlsanity.com/paper/2306.02314/) (2023)
- [Knowledge Transfer with Simulated Inter-Image Erasing for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2407.02768/) (2024)
- [WeakMedSAM: Weakly-Supervised Medical Image Segmentation via SAM with Sub-Class Exploration and Prompt Affinity Mining](https://mlsanity.com/paper/2503.04106/) (2025)
- [Centered Self-Attention Layers](https://mlsanity.com/paper/2306.01610/) (2023)
- [Active Label Correction for Semantic Segmentation with Foundation Models](https://mlsanity.com/paper/2403.10820/) (2024)
- [There is no SAMantics! Exploring SAM as a Backbone for Visual Understanding Tasks](https://mlsanity.com/paper/2411.15288/) (2024)
- [Knowledge Extraction and Distillation from Large-Scale Image-Text Colonoscopy Records Leveraging Large Language and Vision Models](https://mlsanity.com/paper/2310.11173/) (2023)
- [Tuning-free Universally-Supervised Semantic Segmentation](https://mlsanity.com/paper/2405.14294/) (2024)
- [Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentation](https://mlsanity.com/paper/2410.01341/) (2024)
- [G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Models](https://mlsanity.com/paper/2506.01539/) (2025)
- [Tackling Ambiguity from Perspective of Uncertainty Inference and Affinity Diversification for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2404.08195/) (2024)
- [Fine-grained Background Representation for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2406.15755/) (2024)

## Abstract (arXiv)

Weakly-Supervised Semantic Segmentation (WSSS) using image-level labels typically utilizes Class Activation Map (CAM) to generate the pseudo labels. Limited by the local structure perception of CNN, CAM usually cannot identify the integral object regions. Though the recent Vision Transformer (ViT) can remedy this flaw, we observe it also brings the over-smoothing issue, ie, the final patch tokens incline to be uniform. In this work, we propose Token Contrast (ToCo) to address this issue and further explore the virtue of ViT for WSSS. Firstly, motivated by the observation that intermediate layers in ViT can still retain semantic diversity, we designed a Patch Token Contrast module (PTC). PTC supervises the final patch tokens with the pseudo token relations derived from intermediate layers, allowing them to align the semantic regions and thus yield more accurate CAM. Secondly, to further differentiate the low-confidence regions in CAM, we devised a Class Token Contrast module (CTC) inspired by the fact that class tokens in ViT can capture high-level semantics. CTC facilitates the representation consistency between uncertain local regions and global objects by contrasting their class tokens. Experiments on the PASCAL VOC and MS COCO datasets show the proposed ToCo can remarkably surpass other single-stage competitors and achieve comparable performance with state-of-the-art multi-stage methods. Code is available at https://github.com/rulixiang/ToCo.