# AlignSeg: Feature-Aligned Segmentation Networks

arXiv: https://arxiv.org/abs/2003.00872 · HTML: https://mlsanity.com/paper/2003.00872/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2020.

## Built on by (most cited first, 30 of 53)

- [Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer](https://mlsanity.com/paper/2106.03650/) (2021)
- [An Empirical Study of Remote Sensing Pretraining](https://mlsanity.com/paper/2204.02825/) (2022)
- [TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation](https://mlsanity.com/paper/2204.05525/) (2022)
- [FaPN: Feature-aligned Pyramid Network for Dense Image Prediction](https://mlsanity.com/paper/2108.07058/) (2021)
- [Changer: Feature Interaction is What You Need for Change Detection](https://mlsanity.com/paper/2209.08290/) (2022)
- [UltraSR: Spatial Encoding is a Missing Key for Implicit Image Function-based Arbitrary-Scale Super-Resolution](https://mlsanity.com/paper/2103.12716/) (2021)
- [Learning Mask-aware CLIP Representations for Zero-Shot Segmentation](https://mlsanity.com/paper/2310.00240/) (2023)
- [SeMask: Semantically Masked Transformers for Semantic Segmentation](https://mlsanity.com/paper/2112.12782/) (2021)
- [Frequency-aware Feature Fusion for Dense Image Prediction](https://mlsanity.com/paper/2408.12879/) (2024)
- [SuperFusion: Multilevel LiDAR-Camera Fusion for Long-Range HD Map Generation](https://mlsanity.com/paper/2211.15656/) (2022)
- [Hierarchical Feature Alignment Network for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2207.08485/) (2022)
- [PP-LiteSeg: A Superior Real-Time Semantic Segmentation Model](https://mlsanity.com/paper/2204.02681/) (2022)
- [Learning Implicit Feature Alignment Function for Semantic Segmentation](https://mlsanity.com/paper/2206.08655/) (2022)
- [SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process](https://mlsanity.com/paper/2312.12425/) (2023)
- [Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images](https://mlsanity.com/paper/2106.12413/) (2021)
- [Complementing Onboard Sensors with Satellite Map: A New Perspective for HD Map Construction](https://mlsanity.com/paper/2308.15427/) (2023)
- [RBC: Rectifying the Biased Context in Continual Semantic Segmentation](https://mlsanity.com/paper/2203.08404/) (2022)
- [Focus On Details: Online Multi-object Tracking with Diverse Fine-grained Representation](https://mlsanity.com/paper/2302.14589/) (2023)
- [Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2203.15227/) (2022)
- [Prompt-Matched Semantic Segmentation](https://mlsanity.com/paper/2208.10159/) (2022)
- [SeaFormer++: Squeeze-enhanced Axial Transformer for Mobile Visual Recognition](https://mlsanity.com/paper/2301.13156/) (2023)
- [Self-distilled Feature Aggregation for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2209.07088/) (2022)
- [Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2406.00670/) (2024)
- [Stage-Aware Feature Alignment Network for Real-Time Semantic Segmentation of Street Scenes](https://mlsanity.com/paper/2203.04031/) (2022)
- [AttaNet: Attention-Augmented Network for Fast and Accurate Scene Parsing](https://mlsanity.com/paper/2103.05930/) (2021)
- [SFNet: Faster and Accurate Semantic Segmentation via Semantic Flow](https://mlsanity.com/paper/2207.04415/) (2022)
- [Self-Supervised Monocular Depth Estimation with Self-Reference Distillation and Disparity Offset Refinement](https://mlsanity.com/paper/2302.09789/) (2023)
- [Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2104.00875/) (2021)
- [Dense Prediction with Attentive Feature Aggregation](https://mlsanity.com/paper/2111.00770/) (2021)
- [Representation Separation for Semantic Segmentation with Vision Transformers](https://mlsanity.com/paper/2212.13764/) (2022)

## Abstract (arXiv)

Aggregating features in terms of different convolutional blocks or contextual embeddings has been proven to be an effective way to strengthen feature representations for semantic segmentation. However, most of the current popular network architectures tend to ignore the misalignment issues during the feature aggregation process caused by 1) step-by-step downsampling operations, and 2) indiscriminate contextual information fusion. In this paper, we explore the principles in addressing such feature misalignment issues and inventively propose Feature-Aligned Segmentation Networks (AlignSeg). AlignSeg consists of two primary modules, i.e., the Aligned Feature Aggregation (AlignFA) module and the Aligned Context Modeling (AlignCM) module. First, AlignFA adopts a simple learnable interpolation strategy to learn transformation offsets of pixels, which can effectively relieve the feature misalignment issue caused by multiresolution feature aggregation. Second, with the contextual embeddings in hand, AlignCM enables each pixel to choose private custom contextual information in an adaptive manner, making the contextual embeddings aligned better to provide appropriate guidance. We validate the effectiveness of our AlignSeg network with extensive experiments on Cityscapes and ADE20K, achieving new state-of-the-art mIoU scores of 82.6% and 45.95%, respectively. Our source code will be made available.