# TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation

arXiv: https://arxiv.org/abs/2204.05525 · HTML: https://mlsanity.com/paper/2204.05525/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 60 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)

## Built on by (most cited first, 30 of 82)

- [YOLOv10: Real-Time End-to-End Object Detection](https://mlsanity.com/paper/2405.14458/) (2024)
- [EfficientFormer: Vision Transformers at MobileNet Speed](https://mlsanity.com/paper/2206.01191/) (2022)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [RepViT: Revisiting Mobile CNN From ViT Perspective](https://mlsanity.com/paper/2307.09283/) (2023)
- [Rethinking Vision Transformers for MobileNet Size and Speed](https://mlsanity.com/paper/2212.08059/) (2022)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism](https://mlsanity.com/paper/2309.11331/) (2023)
- [SpectFormer: Frequency and Attention is what you need in a Vision Transformer](https://mlsanity.com/paper/2304.06446/) (2023)
- [EdgeSAM: Prompt-In-the-Loop Distillation for SAM](https://mlsanity.com/paper/2312.06660/) (2023)
- [BiViT: Extremely Compressed Binary Vision Transformer](https://mlsanity.com/paper/2211.07091/) (2022)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Context-Guided Spatial Feature Reconstruction for Efficient Semantic Segmentation](https://mlsanity.com/paper/2405.06228/) (2024)
- [DiffI2I: Efficient Diffusion Model for Image-to-Image Translation](https://mlsanity.com/paper/2308.13767/) (2023)
- [Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformers](https://mlsanity.com/paper/2306.02095/) (2023)
- [SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentation](https://mlsanity.com/paper/2312.17071/) (2023)
- [Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentation](https://mlsanity.com/paper/2407.11344/) (2024)
- [SeaFormer++: Squeeze-enhanced Axial Transformer for Mobile Visual Recognition](https://mlsanity.com/paper/2301.13156/) (2023)
- [Beyond Image Super-Resolution for Image Recognition with Task-Driven Perceptual Loss](https://mlsanity.com/paper/2404.01692/) (2024)
- [Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Model](https://mlsanity.com/paper/2406.19369/) (2024)
- [Scattering Vision Transformer: Spectral Mixing Matters](https://mlsanity.com/paper/2311.01310/) (2023)
- [LSNet: See Large, Focus Small](https://mlsanity.com/paper/2503.23135/) (2025)
- [Vision Transformer Adapters for Generalizable Multitask Learning](https://mlsanity.com/paper/2308.12372/) (2023)
- [Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentation](https://mlsanity.com/paper/2407.17261/) (2024)
- [ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attention](https://mlsanity.com/paper/2405.18425/) (2024)
- [Multi-manifold Attention for Vision Transformers](https://mlsanity.com/paper/2207.08569/) (2022)
- [CoFiI2P: Coarse-to-Fine Correspondences for Image-to-Point Cloud Registration](https://mlsanity.com/paper/2309.14660/) (2023)
- [MetaSeg: MetaFormer-based Global Contexts-aware Network for Efficient Semantic Segmentation](https://mlsanity.com/paper/2408.07576/) (2024)
- [MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selection](https://mlsanity.com/paper/2412.16876/) (2024)
- [Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation](https://mlsanity.com/paper/2306.10364/) (2023)

## Abstract (arXiv)

Although vision transformers (ViTs) have achieved great success in computer vision, the heavy computational cost hampers their applications to dense prediction tasks such as semantic segmentation on mobile devices. In this paper, we present a mobile-friendly architecture named Token Pyramid Vision Transformer (TopFormer). The proposed TopFormer takes Tokens from various scales as input to produce scale-aware semantic features, which are then injected into the corresponding tokens to augment the representation. Experimental results demonstrate that our method significantly outperforms CNN- and ViT-based networks across several semantic segmentation datasets and achieves a good trade-off between accuracy and latency. On the ADE20K dataset, TopFormer achieves 5% higher accuracy in mIoU than MobileNetV3 with lower latency on an ARM-based mobile device. Furthermore, the tiny version of TopFormer achieves real-time inference on an ARM-based mobile device with competitive results. The code and models are available at: https://github.com/hustvl/TopFormer