# Affinity Attention Graph Neural Network for Weakly Supervised Semantic Segmentation

arXiv: https://arxiv.org/abs/2106.04054 · HTML: https://mlsanity.com/paper/2106.04054/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 82 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 30 of 53)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1803.10464/) (2018)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations](https://mlsanity.com/paper/1904.05044/) (2019)
- [BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1503.01640/) (2015)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)
- [Simple Does It: Weakly Supervised Instance and Semantic Segmentation](https://mlsanity.com/paper/1603.07485/) (2016)
- [A Simple Pooling-Based Design for Real-Time Salient Object Detection](https://mlsanity.com/paper/1904.09569/) (2019)
- [Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach](https://mlsanity.com/paper/1703.08448/) (2017)
- [Deeply supervised salient object detection with short connections](https://mlsanity.com/paper/1611.04849/) (2016)
- [Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation](https://mlsanity.com/paper/1502.02734/) (2015)
- [Wider or Deeper: Revisiting the ResNet Model for Visual Recognition](https://mlsanity.com/paper/1611.10080/) (2016)
- [FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference](https://mlsanity.com/paper/1902.10421/) (2019)
- [Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1407.1808/) (2014)
- [Weakly-Supervised Semantic Segmentation via Sub-category Exploration](https://mlsanity.com/paper/2008.01183/) (2020)

## Built on by (most cited first, 30 of 47)

- [DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models](https://mlsanity.com/paper/2303.11681/) (2023)
- [Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.02664/) (2022)
- [Graph Foundation Models: Concepts, Opportunities and Challenges](https://mlsanity.com/paper/2310.11829/) (2023)
- [Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02909/) (2022)
- [Cross Language Image Matching for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02668/) (2022)
- [Weakly Supervised Semantic Segmentation using Out-of-Distribution Data](https://mlsanity.com/paper/2203.03860/) (2022)
- [GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs](https://mlsanity.com/paper/2305.12788/) (2023)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Tree Energy Loss: Towards Sparsely Annotated Semantic Segmentation](https://mlsanity.com/paper/2203.10739/) (2022)
- [Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transport](https://mlsanity.com/paper/2308.01779/) (2023)
- [Box2Mask: Box-supervised Instance Segmentation via Level-set Evolution](https://mlsanity.com/paper/2212.01579/) (2022)
- [MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2308.03005/) (2023)
- [A Weakly Supervised Learning Framework for Salient Object Detection via Hybrid Labels](https://mlsanity.com/paper/2209.02957/) (2022)
- [Democracy Does Matter: Comprehensive Feature Mining for Co-Salient Object Detection](https://mlsanity.com/paper/2203.05787/) (2022)
- [Background Activation Suppression for Weakly Supervised Object Localization and Semantic Segmentation](https://mlsanity.com/paper/2309.12943/) (2023)
- [Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2406.11189/) (2024)
- [Adaptive Patch Contrast for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2407.10649/) (2024)
- [Fully and Weakly Supervised Referring Expression Segmentation with End-to-End Learning](https://mlsanity.com/paper/2212.10278/) (2022)
- [Anti-Adversarially Manipulated Attributions for Weakly Supervised Semantic Segmentation and Object Localization](https://mlsanity.com/paper/2204.04890/) (2022)
- [BoxSnake: Polygonal Instance Segmentation with Box Supervision](https://mlsanity.com/paper/2303.11630/) (2023)
- [eX-ViT: A Novel eXplainable Vision Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2207.05358/) (2022)
- [MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2412.11076/) (2024)
- [ZScribbleSeg: Zen and the Art of Scribble Supervised Medical Image Segmentation](https://mlsanity.com/paper/2301.04882/) (2023)
- [Integrating Deep Metric Learning with Coreset for Active Learning in 3D Segmentation](https://mlsanity.com/paper/2411.15763/) (2024)
- [Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns](https://mlsanity.com/paper/2303.11678/) (2023)
- [Label-efficient Segmentation via Affinity Propagation](https://mlsanity.com/paper/2310.10533/) (2023)
- [Dynamic Feature Regularized Loss for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2108.01296/) (2021)
- [SASFormer: Transformers for Sparsely Annotated Semantic Segmentation](https://mlsanity.com/paper/2212.02019/) (2022)
- [GaPro: Box-Supervised 3D Point Cloud Instance Segmentation Using Gaussian Processes as Pseudo Labelers](https://mlsanity.com/paper/2307.13251/) (2023)
- [Hypergraph Convolutional Networks for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2210.05564/) (2022)

## Abstract (arXiv)

Weakly supervised semantic segmentation is receiving great attention due to its low human annotation cost. In this paper, we aim to tackle bounding box supervised semantic segmentation, i.e., training accurate semantic segmentation models using bounding box annotations as supervision. To this end, we propose Affinity Attention Graph Neural Network (A2GNN). Following previous practices, we first generate pseudo semantic-aware seeds, which are then formed into semantic graphs based on our newly proposed affinity Convolutional Neural Network (CNN). Then the built graphs are input to our A2GNN, in which an affinity attention layer is designed to acquire the short- and long- distance information from soft graph edges to accurately propagate semantic labels from the confident seeds to the unlabeled pixels. However, to guarantee the precision of the seeds, we only adopt a limited number of confident pixel seed labels for A2GNN, which may lead to insufficient supervision for training. To alleviate this issue, we further introduce a new loss function and a consistency-checking mechanism to leverage the bounding box constraint, so that more reliable guidance can be included for the model optimization. Experiments show that our approach achieves new state-of-the-art performances on Pascal VOC 2012 datasets (val: 76.5%, test: 75.2%). More importantly, our approach can be readily applied to bounding box supervised instance segmentation task or other weakly supervised semantic segmentation tasks, with state-of-the-art or comparable performance among almot all weakly supervised tasks on PASCAL VOC or COCO dataset. Our source code will be available at https://github.com/zbf1991/A2GNN.