mlsanityData to 14 Oct 2026

MUSTER: A Multi-scale Transformer-based Decoder for Semantic Segmentation

arXiv 2211.13928 ↗First posted 25 Nov 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In recent works on semantic segmentation, there has been a significant focus on designing and integrating transformer-based encoders. However, less attention has been given to transformer-based decoders. We emphasize that the decoder stage is equally vital as the encoder in achieving superior segmentation performance. It disentangles and refines high-level cues, enabling precise object boundary delineation at the pixel level. In this paper, we introduce a novel transformer-based decoder called MUSTER, which seamlessly integrates with hierarchical encoders and consistently delivers high-quality segmentation results, regardless of the encoder architecture. Furthermore, we present a variant of MUSTER that reduces FLOPS while maintaining performance. MUSTER incorporates carefully designed multi-head skip attention (MSKA) units and introduces innovative upsampling operations. The MSKA units enable the fusion of multi-scale features from the encoder and decoder, facilitating comprehensive information integration. The upsampling operation leverages encoder features to enhance object localization and surpasses traditional upsampling methods, improving mIoU (mean Intersection over Union) by 0.4% to 3.2%. On the challenging ADE20K dataset, our best model achieves a single-scale mIoU of 50.23 and a multi-scale mIoU of 51.88, which is on-par with the current state-of-the-art model. Remarkably, we achieve this while significantly reducing the number of FLOPs by 61.3%. Our source code and models are publicly available at: https://github.com/shiwt03/MUSTER.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 64 references that are in the map. First posted on arXiv 25 Nov 2022.

Builds on

38 of 64 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  9. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  12. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
26 more
  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  2. 2016Pyramid Scene Parsing Networkcited by 2,479
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  4. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  5. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
  6. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  7. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  8. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  9. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  10. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  11. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,204
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  13. 2020Image Segmentation Using Deep Learning: A Surveycited by 660
  14. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  15. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  16. 2021Early Convolutions Help Transformers See Bettercited by 396
  17. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  18. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  19. 2021UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformercited by 197
  20. 2019Asymmetric Non-local Neural Networks for Semantic Segmentationcited by 196
  21. 2021Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformercited by 115
  22. 2021Lite Vision Transformer with Enhanced Self-Attentioncited by 64
  23. 2022Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentationcited by 64
  24. 2022SSformer: A Lightweight Transformer for Semantic Segmentationcited by 13
  25. 2020Real-time Semantic Segmentation via Spatial-detail Guided Context Propagationcited by 7
  26. 2022ViT-LSLA: Vision Transformer with Light Self-Limited-Attentioncited by 2

Built on by

Not cited yet.