mlsanityData to 14 Oct 2026

Full Contextual Attention for Multi-resolution Transformers in Semantic Segmentation

arXiv 2212.07890 ↗First posted 15 Dec 2022cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Transformers have proved to be very effective for visual recognition tasks. In particular, vision transformers construct compressed global representations through self-attention and learnable class tokens. Multi-resolution transformers have shown recent successes in semantic segmentation but can only capture local interactions in high-resolution feature maps. This paper extends the notion of global tokens to build GLobal Attention Multi-resolution (GLAM) transformers. GLAM is a generic module that can be integrated into most existing transformer backbones. GLAM includes learnable global tokens, which unlike previous methods can model interactions between all image regions, and extracts powerful representations during training. Extensive experiments show that GLAM-Swin or GLAM-Swin-UNet exhibit substantially better performances than their vanilla counterparts on ADE20K and Cityscapes. Moreover, GLAM can be used to segment large 3D medical images, and GLAM-nnFormer achieves new state-of-the-art performance on the BCV dataset.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

41 of 50 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  9. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  10. 2016Pyramid Scene Parsing Networkcited by 2,479
  11. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
29 more
  1. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  2. 2020Reformer: The Efficient Transformercited by 1,668
  3. 2020Rethinking Attention with Performerscited by 1,601
  4. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  7. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,435
  8. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  9. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  10. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  13. 2018Dual Attention Network for Scene Segmentationcited by 1,003
  14. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  15. 2021Multiscale Vision Transformerscited by 897
  16. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  17. 2018Image Transformercited by 772
  18. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  19. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  20. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 718
  21. 2021Transformer in Transformercited by 633
  22. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  23. 2021Random Feature Attentioncited by 354
  24. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  25. 2020Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformerscited by 190
  26. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  27. 2020Disentangled Non-Local Neural Networkscited by 132
  28. 2020Strip Pooling: Rethinking Spatial Pooling for Scene Parsingcited by 114
  29. 2021TransClaw U-Net: Claw U-Net with Transformers for Medical Image Segmentationcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 0; 2026: 2.

5 papers.

  1. 2023OpenForest: A data catalogue for machine learning in forest monitoringcited by 15
  2. 2024A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationshipscited by 11
  3. 2023TRUSTED: The Paired 3D Transabdominal Ultrasound and CT Human Data for Kidney Segmentation and Registration Researchcited by 3
  4. 2026Context-Aware Semantic Segmentation via Stage-Wise Attentioncited by 1
  5. 2026Adapting Vision Transformers to Ultra-High Resolution Semantic Segmentation with Relay Tokenscited by 0