# Semantic-Constraint Matching Transformer for Weakly Supervised Object Localization

arXiv: https://arxiv.org/abs/2309.01331 · HTML: https://mlsanity.com/paper/2309.01331/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 64 references that are in the map.
- First posted on arXiv 4 Sep 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Structure-measure: A New Way to Evaluate Foreground Maps](https://mlsanity.com/paper/1708.00786/) (2017)
- [Enhanced-alignment Measure for Binary Foreground Map Evaluation](https://mlsanity.com/paper/1805.10421/) (2018)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localization](https://mlsanity.com/paper/1704.04232/) (2017)
- [Visual Saliency Based on Multiscale Deep Features](https://mlsanity.com/paper/1503.08663/) (2015)
- [Attention-based Dropout Layer for Weakly Supervised Object Localization](https://mlsanity.com/paper/1908.10028/) (2019)
- [TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization](https://mlsanity.com/paper/2103.14862/) (2021)
- [Self-produced Guidance for Weakly-supervised Object Localization](https://mlsanity.com/paper/1807.08902/) (2018)
- [Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2103.01795/) (2021)
- [Inter-Image Communication for Weakly Supervised Localization](https://mlsanity.com/paper/2008.05096/) (2020)
- [A Unified Transformer Framework for Group-based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection](https://mlsanity.com/paper/2203.04708/) (2022)
- [Rethinking the Route Towards Weakly Supervised Object Localization](https://mlsanity.com/paper/2002.11359/) (2020)
- [Unveiling the Potential of Structure Preserving for Weakly Supervised Object Localization](https://mlsanity.com/paper/2103.04523/) (2021)
- [Multi-source weak supervision for saliency detection](https://mlsanity.com/paper/1904.00566/) (2019)
- [Geometry Constrained Weakly Supervised Object Localization](https://mlsanity.com/paper/2007.09727/) (2020)
- [Weakly Supervised Salient Object Detection Using Image Labels](https://mlsanity.com/paper/1803.06503/) (2018)
- [MFNet: Multi-filter Directive Network for Weakly Supervised Salient Object Detection](https://mlsanity.com/paper/2112.01732/) (2021)
- [Decoupled Spatial Neural Attention for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1803.02563/) (2018)
- [Improving Weakly-supervised Object Localization via Causal Intervention](https://mlsanity.com/paper/2104.10351/) (2021)
- [Exploiting Web Images for Weakly Supervised Object Detection](https://mlsanity.com/paper/1707.08721/) (2017)
- [Dual Progressive Transformations for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2209.15211/) (2022)

## Built on by (most cited first, 2 of 2)

- [COMNet: Co-Occurrent Matching for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2309.16959/) (2023)
- [Dual-Augmented Transformer Network for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2310.00307/) (2023)

## Abstract (arXiv)

Weakly supervised object localization (WSOL) strives to learn to localize objects with only image-level supervision. Due to the local receptive fields generated by convolution operations, previous CNN-based methods suffer from partial activation issues, concentrating on the object's discriminative part instead of the entire entity scope. Benefiting from the capability of the self-attention mechanism to acquire long-range feature dependencies, Vision Transformer has been recently applied to alleviate the local activation drawbacks. However, since the transformer lacks the inductive localization bias that are inherent in CNNs, it may cause a divergent activation problem resulting in an uncertain distinction between foreground and background. In this work, we proposed a novel Semantic-Constraint Matching Network (SCMN) via a transformer to converge on the divergent activation. Specifically, we first propose a local patch shuffle strategy to construct the image pairs, disrupting local patches while guaranteeing global consistency. The paired images that contain the common object in spatial are then fed into the Siamese network encoder. We further design a semantic-constraint matching module, which aims to mine the co-object part by matching the coarse class activation maps (CAMs) extracted from the pair images, thus implicitly guiding and calibrating the transformer network to alleviate the divergent activation. Extensive experimental results conducted on two challenging benchmarks, including CUB-200-2011 and ILSVRC datasets show that our method can achieve the new state-of-the-art performance and outperform the previous method by a large margin.