mlsanityData to 14 Oct 2026

Segformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation

arXiv 2405.14467 ↗First posted 23 May 2024cs.CV cs.AI cs.LGcited by 9

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Utilizing transformer architectures for semantic segmentation of high-resolution images is hindered by the attention's quadratic computational complexity in the number of tokens. A solution to this challenge involves decreasing the number of tokens through token merging, which has exhibited remarkable enhancements in inference speed, training efficiency, and memory utilization for image classification tasks. In this paper, we explore various token merging strategies within the framework of the Segformer architecture and perform experiments on multiple semantic segmentation and human pose estimation datasets. Notably, without model re-training, we, for example, achieve an inference acceleration of 61% on the Cityscapes dataset while maintaining the mIoU performance. Consequently, this paper facilitates the deployment of transformer-based architectures on resource-constrained devices and in real-time applications.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 28 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

25 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Token Merging: Your ViT But Faster17 Oct
  2. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec

Other references · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  9. 2016Pyramid Scene Parsing Networkcited by 2,479
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  11. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  12. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
11 more
  1. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  2. 2021Multiscale Vision Transformerscited by 897
  3. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  4. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
  5. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  6. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  7. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  8. 2023Token Merging for Fast Stable Diffusioncited by 224
  9. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  10. 2021Dynamically pruning segformer for efficient semantic segmentationcited by 13
  11. 2023All Keypoints You Need: Detecting Arbitrary Keypoints on the Body of Triple, High, and Long Jump Athletescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 4.

9 papers.

  1. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 22
  2. 2025Towards Ball Spin and Trajectory Analysis in Table Tennis Broadcast Videos via Physically Grounded Synthetic-to-Real Transfercited by 4
  3. 2025Uplifting Table Tennis: A Robust, Real-World Application for 3D Trajectory and Spin Estimationcited by 3
  4. 2025Video, How Do Your Tokens Merge?cited by 2
  5. 2026Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrievalcited by 2
  6. 2026TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videoscited by 1
  7. 2025MMMS: Multi-Modal Multi-Surface Interactive Segmentationcited by 0
  8. 2026DSFlash: Comprehensive Panoptic Scene Graph Generation in Realtimecited by 0
  9. 2026Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
  2. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation
  3. 2026Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM
  4. 2024Token Cropr: Faster ViTs for Quite a Few Tasks
  5. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers
  6. 2025ClustViT: Clustering-based Token Merging for Semantic Segmentation
  7. 2024ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition
  8. 2025I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation