mlsanityData to 14 Oct 2026

Adapting Vision Transformers to Ultra-High Resolution Semantic Segmentation with Relay Tokens

arXiv 2601.05927 ↗First posted 9 Jan 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Current approaches for segmenting ultra high resolution images either slide a window, thereby discarding global context, or downsample and lose fine detail. We propose a simple yet effective method that brings explicit multi scale reasoning to vision transformers, simultaneously preserving local details and global awareness. Concretely, we process each image in parallel at a local scale (high resolution, small crops) and a global scale (low resolution, large crops), and aggregate and propagate features between the two branches with a small set of learnable relay tokens. The design plugs directly into standard transformer backbones (eg ViT and Swin) and adds fewer than 2 % parameters. Extensive experiments on three ultra high resolution segmentation benchmarks, Archaeoscape, URUR, and Gleason, and on the conventional Cityscapes dataset show consistent gains, with up to 15 % relative mIoU improvement. Code and pretrained models are available at https://archaeoscape.ai/work/relay-tokens/ .

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 9 Jan 2026.

Builds on

44 of 54 references.

References · 44

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023Segment Anythingcited by 8,234
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  10. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  12. 2016Pyramid Scene Parsing Networkcited by 2,479
32 more
  1. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  2. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  3. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  5. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  6. 2023Vision Transformers Need Registerscited by 871
  7. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  8. 2021ImageNet-21K Pretraining for the Massescited by 613
  9. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 328
  10. 2022Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learningcited by 308
  11. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  12. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  13. 2020Hierarchical Multi-Scale Attention for Semantic Segmentationcited by 183
  14. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 171
  15. 2019Gated-SCNN: Gated Shape CNNs for Semantic Segmentationcited by 166
  16. 2020CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinementcited by 113
  17. 2019Boundary-Aware Feature Propagation for Scene Segmentationcited by 107
  18. 2019Collaborative Global-Local Networks for Memory-Efficient Segmentation of Ultra-High Resolution Imagescited by 51
  19. 2015Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transformcited by 46
  20. 2019Adaptive Weighting Multi-Field-of-View CNN for Semantic Segmentation in Pathologycited by 36
  21. 2021High Quality Segmentation for Ultra High-resolution Imagescited by 31
  22. 2024SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentationcited by 31
  23. 2019Deep Multi-Magnification Networks for Multi-Class Breast Cancer Image Segmentationcited by 22
  24. 2023Ultra-High Resolution Segmentation with Ultra-Rich Context: A Novel Benchmarkcited by 22
  25. 2019Multi-scale fully convolutional neural networks for histopathology image segmentation: from nuclear aberrations to the global tissue architecturecited by 20
  26. 2018Multi-Resolution Networks for Semantic Segmentation in Whole Slide Imagescited by 14
  27. 2024Ultra-High Resolution Segmentation via Boundary-Enhanced Patch-Merging Transformercited by 10
  28. 2024xT: Nested Tokenization for Larger Context in Large Imagescited by 8
  29. 2024Parameter-Inverted Image Pyramid Networkscited by 8
  30. 2022Full Contextual Attention for Multi-resolution Transformers in Semantic Segmentationcited by 5
  31. 2022Memory transformers for full context and high-resolution 3D Medical Segmentationcited by 3
  32. 2024Archaeoscape: Bringing Aerial Laser Scanning Archaeology to the Deep Learning Eracited by 2

Built on by

Not cited yet.