# SASFormer: Transformers for Sparsely Annotated Semantic Segmentation

arXiv: https://arxiv.org/abs/2212.02019 · HTML: https://mlsanity.com/paper/2212.02019/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 5 Dec 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization](https://mlsanity.com/paper/2103.14862/) (2021)
- [On Regularized Losses for Weakly-supervised CNN Segmentation](https://mlsanity.com/paper/1803.09569/) (2018)
- [ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition](https://mlsanity.com/paper/2208.10431/) (2022)
- [Token Shift Transformer for Video Classification](https://mlsanity.com/paper/2108.02432/) (2021)
- [Affinity Attention Graph Neural Network for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2106.04054/) (2021)
- [Tree Energy Loss: Towards Sparsely Annotated Semantic Segmentation](https://mlsanity.com/paper/2203.10739/) (2022)
- [Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning](https://mlsanity.com/paper/2105.00957/) (2021)
- [Weakly Supervised Scene Parsing with Point-based Distance Metric Learning](https://mlsanity.com/paper/1811.02233/) (2018)
- [Cross-Modality High-Frequency Transformer for MR Image Super-Resolution](https://mlsanity.com/paper/2203.15314/) (2022)
- [Scribble-Supervised Semantic Segmentation by Uncertainty Reduction on Neural Representation and Self-Supervision on Neural Eigenspace](https://mlsanity.com/paper/2102.09896/) (2021)
- [Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching](https://mlsanity.com/paper/2112.09459/) (2021)
- [Seminar Learning for Click-Level Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2108.13393/) (2021)
- [Re-Attention Transformer for Weakly Supervised Object Localization](https://mlsanity.com/paper/2208.01838/) (2022)
- [Long-term Leap Attention, Short-term Periodic Shift for Video Classification](https://mlsanity.com/paper/2207.05526/) (2022)

## Built on by (most cited first, 6 of 6)

- [Scribbles for All: Benchmarking Scribble Supervised Segmentation Across Datasets](https://mlsanity.com/paper/2408.12489/) (2024)
- [Calibrating Undisciplined Over-Smoothing in Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.03112/) (2023)
- [Masked Collaborative Contrast for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.08491/) (2023)
- [ClickAttention: Click Region Similarity Guided Interactive Segmentation](https://mlsanity.com/paper/2408.06021/) (2024)
- [Rewis3d: Reconstruction Improves Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2603.06374/) (2026)
- [TACoS: Weakly Supervised Learning of Two-Dimensional Materials from Scribble Annotations to Precise Segmentation](https://mlsanity.com/paper/2607.07169/) (2026)

## Abstract (arXiv)

Semantic segmentation based on sparse annotation has advanced in recent years. It labels only part of each object in the image, leaving the remainder unlabeled. Most of the existing approaches are time-consuming and often necessitate a multi-stage training strategy. In this work, we propose a simple yet effective sparse annotated semantic segmentation framework based on segformer, dubbed SASFormer, that achieves remarkable performance. Specifically, the framework first generates hierarchical patch attention maps, which are then multiplied by the network predictions to produce correlated regions separated by valid labels. Besides, we also introduce the affinity loss to ensure consistency between the features of correlation results and network predictions. Extensive experiments showcase that our proposed approach is superior to existing methods and achieves cutting-edge performance. The source code is available at urlhttps://github.com/su-hui-zz/SASFormer.