mlsanityData to 14 Oct 2026

CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbones

arXiv 2509.21764 ↗First posted 26 Sep 2025cs.CV cs.LGcited by 0

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Many modern ViT backbones adopt spatial architectural designs, such as window attention, decomposed relative positional embeddings in SAM, and RoPE in DINOv3. Such architectures impose new challenges on token reduction, as the vast majority of existing methods fail to preserve the spatial structure these architectures depend on. In this paper, we introduce a simple yet effective token merging method that maintains spatial integrity, enabling seamless compatibility with spatial architectures. We reconcile two seemingly conflicting requirements: (i)exploiting the uneven information distribution across the spatial layout while (ii)preserving the spatial structure post-merging. Our approach employs (i)a 2D reduction strategy to enforce structured token layouts, (ii)a spatial-aware merging algorithm that maintains relative token positions, and (iii)a novel max-magnitude-per-dimension token representation that preserves salient features. Our method demonstrates strong performance both off-the-shelf and with fine-tuning, achieving state-of-the-art results on spatial and non-spatial architectures across various vision tasks. Specifically, we achieve 1.25x speedup on SAM-H with only 0.7% mIOU drop evaluated on COCO off-the-shelf, and 1.15x speedup on DeiT-B with no top-1 accuracy drop on ImageNet within just one epoch of fine-tuning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 37 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

34 of 37 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers13 Dec
  2. 2024Efficient Visual Transformer by Learnable Token Merging21 Jul
  3. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers14 Jun
  4. 2024Accelerating Transformers with Spectrum-Preserving Token Merging25 May
  5. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging2 Dec
  6. 2023GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation6 Nov
  7. 2023Revisiting Token Pruning for Object Detection and Instance Segmentation12 Jun
  8. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformers29 May
  9. 2022Token Merging: Your ViT But Faster17 Oct
  10. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec

Other references · 24

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2023Segment Anythingcited by 8,234
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  7. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  8. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  9. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  10. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  11. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  12. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
12 more
  1. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 290
  2. 2024Rotary Position Embedding for Vision Transformercited by 236
  3. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 181
  4. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  5. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  6. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  7. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 40
  8. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  9. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  10. 2024Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformerscited by 26
  11. 2023Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformerscited by 22
  12. 2023AiluRus: A Scalable ViT Framework for Dense Predictioncited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Training-Free Acceleration of ViTs with Delayed Spatial Merging
  2. 2026MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
  3. 2026StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models
  4. 2025FastVGGT: Training-Free Acceleration of Visual Geometry Transformer
  5. 2025Neighbor-Aware Token Reduction via Hilbert Curve for Vision Transformers
  6. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
  7. 2025Sequential Token Merging: Revisiting Hidden States
  8. 2026When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compression