# StructToken : Rethinking Semantic Segmentation with Structural Prior

arXiv: https://arxiv.org/abs/2203.12612 · HTML: https://mlsanity.com/paper/2203.12612/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 66 references that are in the map.
- First posted on arXiv 23 Mar 2022.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation](https://mlsanity.com/paper/1901.02985/) (2019)

## Built on by (most cited first, 30 of 30)

- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [DDP: Diffusion Model for Dense Visual Prediction](https://mlsanity.com/paper/2303.17559/) (2023)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [SegViT: Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2210.05844/) (2022)
- [SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2306.06289/) (2023)
- [UniNeXt: Exploring A Unified Architecture for Vision Recognition](https://mlsanity.com/paper/2304.13700/) (2023)
- [Prototype-Driven Structure Synergy Network for Remote Sensing Images Segmentation](https://mlsanity.com/paper/2508.04022/) (2025)
- [Bring Adaptive Binding Prototypes to Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2405.15169/) (2024)
- [Representation Separation for Semantic Segmentation with Vision Transformers](https://mlsanity.com/paper/2212.13764/) (2022)
- [PRSeg: A Lightweight Patch Rotate MLP Decoder for Semantic Segmentation](https://mlsanity.com/paper/2305.00671/) (2023)
- [Deep Common Feature Mining for Efficient Video Semantic Segmentation](https://mlsanity.com/paper/2403.02689/) (2024)
- [DSNet: A Novel Way to Use Atrous Convolutions in Semantic Segmentation](https://mlsanity.com/paper/2406.03702/) (2024)
- [Fine-grained Background Representation for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2406.15755/) (2024)
- [Category Feature Transformer for Semantic Segmentation](https://mlsanity.com/paper/2308.05581/) (2023)
- [Boosting Semantic Segmentation from the Perspective of Explicit Class Embeddings](https://mlsanity.com/paper/2308.12894/) (2023)
- [Minimalist and High-Performance Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2310.12755/) (2023)
- [Domain-invariant Prototypes for Semantic Segmentation](https://mlsanity.com/paper/2208.06087/) (2022)
- [CARD: Semantic Segmentation with Efficient Class-Aware Regularized Decoder](https://mlsanity.com/paper/2301.04258/) (2023)
- [High-level Feature Guided Decoding for Semantic Segmentation](https://mlsanity.com/paper/2303.08646/) (2023)
- [Exploring vision transformer layer choosing for semantic segmentation](https://mlsanity.com/paper/2305.01279/) (2023)
- [AxWin Transformer: A Context-Aware Vision Transformer Backbone with Axial Windows](https://mlsanity.com/paper/2305.01280/) (2023)
- [VPNeXt -- Rethinking Dense Decoding for Plain Vision Transformer](https://mlsanity.com/paper/2502.16654/) (2025)
- [Segment Any Vehicle: Semantic and Visual Context Driven SAM and A Benchmark](https://mlsanity.com/paper/2508.04260/) (2025)
- [Deep Learning for Crack Detection: A Review of Learning Paradigms, Generalizability, and Datasets](https://mlsanity.com/paper/2508.10256/) (2025)
- [I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation](https://mlsanity.com/paper/2509.10334/) (2025)
- [Towards Customized Knowledge Distillation for Chip-Level Dense Image Predictions](https://mlsanity.com/paper/2401.13174/) (2024)
- [Is Pre-training Applicable to the Decoder for Dense Prediction?](https://mlsanity.com/paper/2503.07637/) (2025)
- [Classifier Enhancement Using Extended Context and Domain Experts for Semantic Segmentation](https://mlsanity.com/paper/2510.25174/) (2025)
- [Token-Space Mask Prediction for Efficient Vision Transformer Segmentation](https://mlsanity.com/paper/2605.18177/) (2026)
- [Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM](https://mlsanity.com/paper/2609.18279/) (2026)

## Abstract (arXiv)

In previous deep-learning-based methods, semantic segmentation has been regarded as a static or dynamic per-pixel classification task, textiti.e., classify each pixel representation to a specific category. However, these methods only focus on learning better pixel representations or classification kernels while ignoring the structural information of objects, which is critical to human decision-making mechanism. In this paper, we present a new paradigm for semantic segmentation, named structure-aware extraction. Specifically, it generates the segmentation results via the interactions between a set of learned structure tokens and the image feature, which aims to progressively extract the structural information of each category from the feature. Extensive experiments show that our StructToken outperforms the state-of-the-art on three widely-used benchmarks, including ADE20K, Cityscapes, and COCO-Stuff-10K.