# RTFormer: Efficient Design for Real-Time Semantic Segmentation with Transformer

arXiv: https://arxiv.org/abs/2210.07124 · HTML: https://mlsanity.com/paper/2210.07124/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 58 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)

## Built on by (most cited first, 30 of 31)

- [Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformers](https://mlsanity.com/paper/2306.02095/) (2023)
- [SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentation](https://mlsanity.com/paper/2312.17071/) (2023)
- [Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentation](https://mlsanity.com/paper/2407.11344/) (2024)
- [Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustness](https://mlsanity.com/paper/2503.18445/) (2025)
- [Mask-FPAN: Semi-Supervised Face Parsing in the Wild With De-Occlusion and UV GAN](https://mlsanity.com/paper/2212.09098/) (2022)
- [Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentation](https://mlsanity.com/paper/2407.17261/) (2024)
- [MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selection](https://mlsanity.com/paper/2412.16876/) (2024)
- [MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for Multi-modal Semantic Segmentation](https://mlsanity.com/paper/2503.06700/) (2025)
- [SituationAdapt: Contextual UI Optimization in Mixed Reality with Situation Awareness via LLM Reasoning](https://mlsanity.com/paper/2409.12836/) (2024)
- [SERNet-Former: Semantic Segmentation by Efficient Residual Network with Attention-Boosting Gates and Attention-Fusion Networks](https://mlsanity.com/paper/2401.15741/) (2024)
- [PP-MobileSeg: Explore the Fast and Accurate Semantic Segmentation Model on Mobile Devices](https://mlsanity.com/paper/2304.05152/) (2023)
- [Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided Dynamic Token Merge for Document Understanding](https://mlsanity.com/paper/2305.11392/) (2023)
- [Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysis](https://mlsanity.com/paper/2403.04080/) (2024)
- [AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation](https://mlsanity.com/paper/2408.01708/) (2024)
- [Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation](https://mlsanity.com/paper/2407.13254/) (2024)
- [Communicate Less, Synthesize the Rest: Latency-aware Intent-based Generative Semantic Multicasting with Diffusion Models](https://mlsanity.com/paper/2411.02334/) (2024)
- [Revisiting Efficient Semantic Segmentation: Learning Offsets for Better Spatial and Class Feature Alignment](https://mlsanity.com/paper/2508.08811/) (2025)
- [Principles of Forgetting in Domain-Incremental Semantic Segmentation in Adverse Weather Conditions](https://mlsanity.com/paper/2303.14115/) (2023)
- [EFormer: Enhanced Transformer towards Semantic-Contour Features of Foreground for Portraits Matting](https://mlsanity.com/paper/2308.12831/) (2023)
- [Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights](https://mlsanity.com/paper/2403.19882/) (2024)
- [DSNet: A Novel Way to Use Atrous Convolutions in Semantic Segmentation](https://mlsanity.com/paper/2406.03702/) (2024)
- [RT-K-Net: Revisiting K-Net for Real-Time Panoptic Segmentation](https://mlsanity.com/paper/2305.01255/) (2023)
- [Bilateral Network with Residual U-blocks and Dual-Guided Attention for Real-time Semantic Segmentation](https://mlsanity.com/paper/2310.20305/) (2023)
- [A feature refinement module for light-weight semantic segmentation network](https://mlsanity.com/paper/2412.08670/) (2024)
- [HDBFormer: Efficient RGB-D Semantic Segmentation with A Heterogeneous Dual-Branch Framework](https://mlsanity.com/paper/2504.13579/) (2025)
- [TSLA: A Task-Specific Learning Adaptation for Semantic Segmentation on Autonomous Vehicles Platform](https://mlsanity.com/paper/2508.12279/) (2025)
- [A Decoding Scheme with Successive Aggregation of Multi-Level Features for Light-Weight Semantic Segmentation](https://mlsanity.com/paper/2402.11201/) (2024)
- [MGNiceNet: Unified Monocular Geometric Scene Understanding](https://mlsanity.com/paper/2411.11466/) (2024)
- [WaveSeg: Enhancing Segmentation Precision via High-Frequency Prior and Mamba-Driven Spectrum Decomposition](https://mlsanity.com/paper/2510.21079/) (2025)
- [Cross-Stage Attention Propagation for Efficient Semantic Segmentation](https://mlsanity.com/paper/2604.05431/) (2026)

## Abstract (arXiv)

Recently, transformer-based networks have shown impressive results in semantic segmentation. Yet for real-time semantic segmentation, pure CNN-based approaches still dominate in this field, due to the time-consuming computation mechanism of transformer. We propose RTFormer, an efficient dual-resolution transformer for real-time semantic segmenation, which achieves better trade-off between performance and efficiency than CNN-based models. To achieve high inference efficiency on GPU-like devices, our RTFormer leverages GPU-Friendly Attention with linear complexity and discards the multi-head mechanism. Besides, we find that cross-resolution attention is more efficient to gather global context information for high-resolution branch by spreading the high level knowledge learned from low-resolution branch. Extensive experiments on mainstream benchmarks demonstrate the effectiveness of our proposed RTFormer, it achieves state-of-the-art on Cityscapes, CamVid and COCOStuff, and shows promising results on ADE20K. Code is available at PaddleSeg: https://github.com/PaddlePaddle/PaddleSeg.