# Transformer Scale Gate for Semantic Segmentation

arXiv: https://arxiv.org/abs/2205.07056 · HTML: https://mlsanity.com/paper/2205.07056/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 14 May 2022.

## Builds on (most cited first, 30 of 35)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [Attention to Scale: Scale-aware Semantic Image Segmentation](https://mlsanity.com/paper/1511.03339/) (2015)
- [Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2105.15075/) (2021)
- [MPViT: Multi-Path Vision Transformer for Dense Prediction](https://mlsanity.com/paper/2112.11010/) (2021)
- [Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation](https://mlsanity.com/paper/2111.01236/) (2021)
- [P2T: Pyramid Pooling Transformer for Scene Understanding](https://mlsanity.com/paper/2106.12011/) (2021)
- [GFF: Gated Fully Fusion for Semantic Segmentation](https://mlsanity.com/paper/1904.01803/) (2019)

## Built on by (most cited first, 17 of 17)

- [ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2406.09936/) (2024)
- [Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts](https://mlsanity.com/paper/2412.04220/) (2024)
- [Open-Vocabulary Object Detection via Scene Graph Discovery](https://mlsanity.com/paper/2307.03339/) (2023)
- [Soft labelling for semantic segmentation: Bringing coherence to label down-sampling](https://mlsanity.com/paper/2302.13961/) (2023)
- [Unified Open-Vocabulary Dense Visual Prediction](https://mlsanity.com/paper/2307.08238/) (2023)
- [High-level Feature Guided Decoding for Semantic Segmentation](https://mlsanity.com/paper/2303.08646/) (2023)
- [FANet: Feature Amplification Network for Semantic Segmentation in Cluttered Background](https://mlsanity.com/paper/2407.09379/) (2024)
- [Image Recognition with Online Lightweight Vision Transformer: A Survey](https://mlsanity.com/paper/2505.03113/) (2025)
- [Large coordinate kernel attention network for lightweight image super-resolution](https://mlsanity.com/paper/2405.09353/) (2024)
- [VPNeXt -- Rethinking Dense Decoding for Plain Vision Transformer](https://mlsanity.com/paper/2502.16654/) (2025)
- [Optimized Information Flow for Transformer Tracking](https://mlsanity.com/paper/2402.08195/) (2024)
- [Hybrid Deep Learning-Based for Enhanced Occlusion Segmentation in PICU Patient Monitoring](https://mlsanity.com/paper/2407.13341/) (2024)
- [Efficient Star Distillation Attention Network for Lightweight Image Super-Resolution](https://mlsanity.com/paper/2506.12475/) (2025)
- [MPT: Motion Prompt Tuning for Micro-Expression Recognition](https://mlsanity.com/paper/2508.09446/) (2025)
- [Classifier Enhancement Using Extended Context and Domain Experts for Semantic Segmentation](https://mlsanity.com/paper/2510.25174/) (2025)
- [Fluxamba: Topology-Aware Anisotropic State Space Models for Geological Lineament Segmentation in Multi-Source Remote Sensing](https://mlsanity.com/paper/2601.17288/) (2026)
- [Contextrast++: Robust Multi-Scale Contextual Contrastive Learning for Semantic Segmentation](https://mlsanity.com/paper/2608.22679/) (2026)

## Abstract (arXiv)

Effectively encoding multi-scale contextual information is crucial for accurate semantic segmentation. Existing transformer-based segmentation models combine features across scales without any selection, where features on sub-optimal scales may degrade segmentation outcomes. Leveraging from the inherent properties of Vision Transformers, we propose a simple yet effective module, Transformer Scale Gate (TSG), to optimally combine multi-scale features.TSG exploits cues in self and cross attentions in Vision Transformers for the scale selection. TSG is a highly flexible plug-and-play module, and can easily be incorporated with any encoder-decoder-based hierarchical vision Transformer architecture. Extensive experiments on the Pascal Context and ADE20K datasets demonstrate that our feature selection strategy achieves consistent gains.