Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM
On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs
Abstract (arXiv)
Vision Transformers (ViTs) have achieved state-of-the-art performance across a range of computer vision tasks, mainly thanks to the self-attention mechanism. However, its complexity, increasing quadratically with the number of tokens, remains the major obstacle to ViT efficiency and deployment at scale. Token merging reduces this cost by aggregating redundant tokens. Yet existing methods are typically evaluated within a single architecture, leaving open whether their effectiveness stems from the merging mechanism itself or from the specific decoder they are paired with. We extend Graph-Guided Token Merging (G2TM), a single module inserted early in a ViT-based network, beyond its original Segmenter setting. We evaluate G2TM across three semantic segmentation frameworks (Segmenter, SETR, EoMT) and three decoder families (Linear, Transformer-, convolution-based), as well as standard ViT image classification. Our results show that G2TM's behavior and accuracy-efficiency trade-off are consistent across every tested architecture for a given backbone size, indicating that its effectiveness is a property of the encoder rather than the decoder. G2TM also generalizes well to image classification, achieving an even smaller degradation in accuracy compared to semantic segmentation. We further find that G2TM's optimal hyperparameters, resulting in a consistent drop in GFLOPs of 22-47% and an increase in throughput by up to 74% for segmentation models on ADE20K dataset, depend primarily on the backbone's pre-training recipe and on the target dataset, rather than on the decoder choice.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 47 references that are in the map. First posted on arXiv 16 Sep 2026.
Builds on
36 of 47 references.
Closest ancestors · 13
Same direction, newest first
- 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI13 Jul
- 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers13 Dec
- 2024Token Cropr: Faster ViTs for Quite a Few Tasks1 Dec
- 2024Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edge1 Jul
- 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers14 Jun
- 2024Accelerating Transformers with Spectrum-Preserving Token Merging25 May
- 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging2 Dec
- 2023GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation6 Nov
- 2023Learned Thresholds Token Merging and Pruning for Vision Transformers20 Jul
- 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformers29 May
- 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers27 May
- 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers15 Nov
1 more
- 2022Token Merging: Your ViT But Faster17 Oct
Other references · 23
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
- 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
- 2021Vision Transformers for Dense Predictioncited by 1,803
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
- 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
- 2023Vision Transformers Need Registerscited by 871
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
- 2021Segmenter: Transformer for Semantic Segmentationcited by 855
- 2021ImageNet-21K Pretraining for the Massescited by 613
11 more
- 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
- 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
- 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 106
- 2025Your ViT is Secretly an Image Segmentation Modelcited by 63
- 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
- 2022StructToken : Rethinking Semantic Segmentation with Structural Priorcited by 30
- 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 29
- 2024Agglomerative Token Clusteringcited by 13
- 2023AiluRus: A Scalable ViT Framework for Dense Predictioncited by 10
- 2023Dynamic Token-Pass Transformers for Semantic Segmentationcited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
- 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
- 2026MergeOver: Post-Training Token Merging for Recursive Vision Transformers
- 2026StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models
- 2024ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition
- 2026RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
- 2023MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers
- 2025ClustViT: Clustering-based Token Merging for Semantic Segmentation