# Head-Free Lightweight Semantic Segmentation with Linear Transformer

arXiv: https://arxiv.org/abs/2301.04648 · HTML: https://mlsanity.com/paper/2301.04648/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 11 Jan 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues](https://mlsanity.com/paper/2007.09355/) (2020)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks](https://mlsanity.com/paper/1905.13545/) (2019)
- [Global Filter Networks for Image Classification](https://mlsanity.com/paper/2107.00645/) (2021)
- [Mobile-Former: Bridging MobileNet and Transformer](https://mlsanity.com/paper/2108.05895/) (2021)
- [Co-Scale Conv-Attentional Image Transformers](https://mlsanity.com/paper/2104.06399/) (2021)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [MPViT: Multi-Path Vision Transformer for Dense Prediction](https://mlsanity.com/paper/2112.11010/) (2021)
- [Shunted Self-Attention via Multi-Scale Token Aggregation](https://mlsanity.com/paper/2111.15193/) (2021)

## Built on by (most cited first, 22 of 22)

- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentation](https://mlsanity.com/paper/2312.17071/) (2023)
- [Event-guided Low-light Video Semantic Segmentation](https://mlsanity.com/paper/2411.00639/) (2024)
- [When Semantic Segmentation Meets Frequency Aliasing](https://mlsanity.com/paper/2403.09065/) (2024)
- [DDLNet: Boosting Remote Sensing Change Detection with Dual-Domain Learning](https://mlsanity.com/paper/2406.13606/) (2024)
- [Mobile-Seed: Joint Semantic Segmentation and Boundary Detection for Mobile Robots](https://mlsanity.com/paper/2311.12651/) (2023)
- [Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures](https://mlsanity.com/paper/2405.18524/) (2024)
- [Task-Generalized Adaptive Cross-Domain Learning for Multimodal Image Fusion](https://mlsanity.com/paper/2508.15505/) (2025)
- [SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation](https://mlsanity.com/paper/2405.06525/) (2024)
- [DSNet: A Novel Way to Use Atrous Convolutions in Semantic Segmentation](https://mlsanity.com/paper/2406.03702/) (2024)
- [CDXLSTM: Boosting Remote Sensing Change Detection with Extended Long Short-Term Memory](https://mlsanity.com/paper/2411.07863/) (2024)
- [Triple-View Knowledge Distillation for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2309.12557/) (2023)
- [Efficient Semantic Segmentation via Lightweight Multiple-Information Interaction Network](https://mlsanity.com/paper/2410.02224/) (2024)
- [Vehicular Road Crack Detection with Deep Learning: A New Online Benchmark for Comprehensive Evaluation of Existing Algorithms](https://mlsanity.com/paper/2503.18082/) (2025)
- [Exploring Lip Segmentation Techniques in Computer Vision: A Comparative Analysis](https://mlsanity.com/paper/2311.11992/) (2023)
- [Dual-Context Aggregation for Universal Image Matting](https://mlsanity.com/paper/2402.18109/) (2024)
- [HyCTAS: Multi-Objective Hybrid Convolution-Transformer Architecture Search for Real-Time Image Segmentation](https://mlsanity.com/paper/2403.10413/) (2024)
- [BoundMatch: Boundary detection applied to semi-supervised segmentation](https://mlsanity.com/paper/2503.23519/) (2025)
- [Vision-Language Feature Alignment for Road Anomaly Segmentation](https://mlsanity.com/paper/2603.01029/) (2026)
- [LiPS: Lightweight Panoptic Segmentation for Resource-Constrained Robotics](https://mlsanity.com/paper/2604.00634/) (2026)
- [Token-Space Mask Prediction for Efficient Vision Transformer Segmentation](https://mlsanity.com/paper/2605.18177/) (2026)
- [Towards Effective Waste Segmentation for Automated Waste Recycling in Cluttered Background](https://mlsanity.com/paper/2606.13587/) (2026)

## Abstract (arXiv)

Existing semantic segmentation works have been mainly focused on designing effective decoders; however, the computational load introduced by the overall structure has long been ignored, which hinders their applications on resource-constrained hardwares. In this paper, we propose a head-free lightweight architecture specifically for semantic segmentation, named Adaptive Frequency Transformer. It adopts a parallel architecture to leverage prototype representations as specific learnable local descriptions which replaces the decoder and preserves the rich image semantics on high-resolution features. Although removing the decoder compresses most of the computation, the accuracy of the parallel structure is still limited by low computational resources. Therefore, we employ heterogeneous operators (CNN and Vision Transformer) for pixel embedding and prototype representations to further save computational costs. Moreover, it is very difficult to linearize the complexity of the vision Transformer from the perspective of spatial domain. Due to the fact that semantic segmentation is very sensitive to frequency information, we construct a lightweight prototype learning block with adaptive frequency filter of complexity O(n) to replace standard self attention with O(n2). Extensive experiments on widely adopted datasets demonstrate that our model achieves superior accuracy while retaining only 3M parameters. On the ADE20K dataset, our model achieves 41.8 mIoU and 4.6 GFLOPs, which is 4.4 mIoU higher than Segformer, with 45% less GFLOPs. On the Cityscapes dataset, our model achieves 78.7 mIoU and 34.4 GFLOPs, which is 2.5 mIoU higher than Segformer with 72.5% less GFLOPs. Code is available at https://github.com/dongbo811/AFFormer.