# Dynamically pruning segformer for efficient semantic segmentation

arXiv: https://arxiv.org/abs/2111.09499 · HTML: https://mlsanity.com/paper/2111.09499/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 23 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [Dynamic Channel Pruning: Feature Boosting and Suppression](https://mlsanity.com/paper/1810.05331/) (2018)

## Built on by (most cited first, 13 of 13)

- [Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies](https://mlsanity.com/paper/2503.02891/) (2025)
- [TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation](https://mlsanity.com/paper/2202.13393/) (2022)
- [HyperspectralViTs: General Hyperspectral Models for On-board Remote Sensing](https://mlsanity.com/paper/2410.17248/) (2024)
- [Segformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/2405.14467/) (2024)
- [WHC: Weighted Hybrid Criterion for Filter Pruning on Convolutional Neural Networks](https://mlsanity.com/paper/2302.08185/) (2023)
- [Class Based Thresholding in Early Exit Semantic Segmentation Networks](https://mlsanity.com/paper/2210.15621/) (2022)
- [DUDA: Distilled Unsupervised Domain Adaptation for Lightweight Semantic Segmentation](https://mlsanity.com/paper/2504.09814/) (2025)
- [Comb, Prune, Distill: Towards Unified Pruning for Vision Model Compression](https://mlsanity.com/paper/2408.03046/) (2024)
- [Optimization of DNN-based HSI Segmentation FPGA-based SoC for ADS: A Practical Approach](https://mlsanity.com/paper/2507.16556/) (2025)
- [TSLA: A Task-Specific Learning Adaptation for Semantic Segmentation on Autonomous Vehicles Platform](https://mlsanity.com/paper/2508.12279/) (2025)
- [Enabling Plant Phenotyping in Weedy Environments using Multi-Modal Imagery via Synthetic and Generated Training Data](https://mlsanity.com/paper/2509.19208/) (2025)
- [ArmFormer: Lightweight Transformer Architecture for Real-Time Multi-Class Weapon Segmentation and Classification](https://mlsanity.com/paper/2510.16854/) (2025)
- [StaticSegFormer: An Efficient High-Performance Semantic Segmentation Based on Static Structured Pruning](https://mlsanity.com/paper/2608.04811/) (2026)

## Abstract (arXiv)

As one of the successful Transformer-based models in computer vision tasks, SegFormer demonstrates superior performance in semantic segmentation. Nevertheless, the high computational cost greatly challenges the deployment of SegFormer on edge devices. In this paper, we seek to design a lightweight SegFormer for efficient semantic segmentation. Based on the observation that neurons in SegFormer layers exhibit large variances across different images, we propose a dynamic gated linear layer, which prunes the most uninformative set of neurons based on the input instance. To improve the dynamically pruned SegFormer, we also introduce two-stage knowledge distillation to transfer the knowledge within the original teacher to the pruned student network. Experimental results show that our method can significantly reduce the computation overhead of SegFormer without an apparent performance drop. For instance, we can achieve 36.9% mIoU with only 3.3G FLOPs on ADE20K, saving more than 60% computation with the drop of only 0.5% in mIoU