# A Decoding Scheme with Successive Aggregation of Multi-Level Features for Light-Weight Semantic Segmentation

arXiv: https://arxiv.org/abs/2402.11201 · HTML: https://mlsanity.com/paper/2402.11201/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 17 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer](https://mlsanity.com/paper/2106.03650/) (2021)
- [In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Images](https://mlsanity.com/paper/1903.08469/) (2019)
- [Lite Vision Transformer with Enhanced Self-Attention](https://mlsanity.com/paper/2112.10809/) (2021)
- [Learning Implicit Feature Alignment Function for Semantic Segmentation](https://mlsanity.com/paper/2206.08655/) (2022)
- [RTFormer: Efficient Design for Real-Time Semantic Segmentation with Transformer](https://mlsanity.com/paper/2210.07124/) (2022)
- [Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention](https://mlsanity.com/paper/2201.01615/) (2022)
- [Cross-Dataset Collaborative Learning for Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2103.11351/) (2021)
- [Efficient Self-Ensemble for Semantic Segmentation](https://mlsanity.com/paper/2111.13280/) (2021)
- [Improving Semantic Segmentation in Transformers using Hierarchical Inter-Level Attention](https://mlsanity.com/paper/2207.02126/) (2022)

## Abstract (arXiv)

Multi-scale architecture, including hierarchical vision transformer, has been commonly applied to high-resolution semantic segmentation to deal with computational complexity with minimum performance loss. In this paper, we propose a novel decoding scheme for semantic segmentation in this regard, which takes multi-level features from the encoder with multi-scale architecture. The decoding scheme based on a multi-level vision transformer aims to achieve not only reduced computational expense but also higher segmentation accuracy, by introducing successive cross-attention in aggregation of the multi-level features. Furthermore, a way to enhance the multi-level features by the aggregated semantics is proposed. The effort is focused on maintaining the contextual consistency from the perspective of attention allocation and brings improved performance with significantly lower computational cost. Set of experiments on popular datasets demonstrates superiority of the proposed scheme to the state-of-the-art semantic segmentation models in terms of computational cost without loss of accuracy, and extensive ablation studies prove the effectiveness of ideas proposed.