mlsanityData to 7 Oct 2026

ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers

arXiv 2406.09936 ↗First posted 14 Jun 2024cs.CVcited by 28

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

This work presents Adaptive Local-then-Global Merging (ALGM), a token reduction method for semantic segmentation networks that use plain Vision Transformers. ALGM merges tokens in two stages: (1) In the first network layer, it merges similar tokens within a small local window and (2) halfway through the network, it merges similar tokens across the entire image. This is motivated by an analysis in which we found that, in those situations, tokens with a high cosine similarity can likely be merged without a drop in segmentation quality. With extensive experiments across multiple datasets and network configurations, we show that ALGM not only significantly improves the throughput by up to 100%, but can also enhance the mean IoU by up to +1.1, thereby achieving a better trade-off between segmentation quality and efficiency than existing methods. Moreover, our approach is adaptive during inference, meaning that the same model can be used for optimal efficiency or accuracy, depending on the application. Code is available at https://tue-mps.github.io/ALGM.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 51 of its 57 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

51 of 57 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers3 Oct
  2. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformers9 Aug
  3. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation2 Aug
  4. 2023Learned Thresholds Token Merging and Pruning for Vision Transformers20 Jul
  5. 2023Revisiting Token Pruning for Object Detection and Instance Segmentation12 Jun
  6. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformers29 May
  7. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers27 May
  8. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers21 Apr
  9. 2022Token Merging: Your ViT But Faster17 Oct
  10. 2022SaiT: Sparse Vision Transformers through Adaptive Token Pruning11 Oct
  11. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec

Other references · 40

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2023Segment Anythingcited by 8,158
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  8. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  9. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  10. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  11. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  12. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
28 more
  1. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  2. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  3. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  4. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  5. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  6. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  7. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  8. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  9. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  10. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  11. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  12. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  13. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 105
  14. 2022Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformercited by 99
  15. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  16. 2022Learning to Merge Tokens in Vision Transformerscited by 64
  17. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  18. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Modelscited by 53
  19. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  20. 2022Vision Transformer with Super Token Samplingcited by 43
  21. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  22. 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 28
  23. 2021Self-slimmed Vision Transformercited by 27
  24. 2023Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformerscited by 22
  25. 2022Transformer Scale Gate for Semantic Segmentationcited by 17
  26. 2023AiluRus: A Scalable ViT Framework for Dense Predictioncited by 10
  27. 2023Dynamic Token-Pass Transformers for Semantic Segmentationcited by 7
  28. 2023PAUMER: Patch Pausing Transformer for Semantic Segmentationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 15.

28 papers.

Sort:

By area:

  1. 2025Your ViT is Secretly an Image Segmentation Modelcited by 61
  2. 2025Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Accelerationcited by 16
  3. 2025Mitigating Semantic Collapse in Partially Relevant Video Retrievalcited by 5
  4. 2025How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?cited by 4
  5. 2026PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoderscited by 3
  6. 2026Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrievalcited by 2
  7. 2026StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Modelscited by 2
  8. 2026MPM: Mutual Pair Merging for Efficient Vision Transformerscited by 2
  9. 2025VPNeXt -- Rethinking Dense Decoding for Plain Vision Transformercited by 1
  10. 2025I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentationcited by 1
  11. 2025What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Mergingcited by 1
  12. 2026Shot-Aware Frame Sampling for Video Understandingcited by 1
16 more
  1. 2025TCSAFormer: Efficient Vision Transformer with Token Compression and Sparse Attention for Medical Image Segmentationcited by 0
  2. 2025Temporal Cluster Assignment for Efficient Real-Time Video Segmentationcited by 0
  3. 2025Towards Efficient Vision State Space Models via Token Mergingcited by 0
  4. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutionscited by 0
  5. 2025CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbonescited by 0
  6. 2025Token Merging via Spatiotemporal Information Mining for Surgical Video Understandingcited by 0
  7. 2026LiPS: Lightweight Panoptic Segmentation for Resource-Constrained Roboticscited by 0
  8. 2026Certainty Is Redundant: Token Sparsification for Efficient Camouflaged Object Detection with Vision Foundation Modelscited by 0
  9. 2026Make Your LVLM KV Cache More Lightweightcited by 0
  10. 2026When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compressioncited by 0
  11. 2026BMFA: Boundary-Minority Free-Energy Adaptive Screeningcited by 0
  12. 2026Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2cited by 0
  13. 2026How Merge-Tolerant Are Vision Transformers for Wheat Phenotyping?cited by 0
  14. 2026Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TMcited by 0
  15. 2026When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Modelcited by 0
  16. 2026Task-Relevant Null-Space Residuals for Non-Injective Neural Mappingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ClustViT: Clustering-based Token Merging for Semantic Segmentation
  2. 2024Segformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation
  3. 2024Token Cropr: Faster ViTs for Quite a Few Tasks
  4. 2025Lossless Token Merging Even Without Fine-Tuning in Vision Transformers
  5. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
  6. 2024Subobject-level Image Tokenization
  7. 2023MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers
  8. 2024Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens