mlsanityData to 7 Oct 2026

StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models

arXiv 2603.07307 ↗First posted 7 Mar 2026cs.CV cs.LGcited by 2

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Recent token merging techniques for Vision Transformers (ViTs) provide substantial speedups by reducing the number of tokens processed by self-attention, often without retraining. However, their direct application to the Segment Anything Model (SAM) family is nontrivial: SAM's image encoder mixes windowed and global attention, and its mask decoder relies on dense, prompt-conditioned features for precise boundary prediction. We systematically evaluate representative token-merging methods on SAM and Medical SAM in a strict off-the-shelf setting, and find that existing destination-selection heuristics can erode boundaries and leak prompt information as merge rates increase. We propose StructSAM, a resolution-preserving merge-unmerge framework tailored to SAM. StructSAM computes a lightweight token-energy score from first-order feature gradients, uses grid-based flatness screening to protect boundary and prompt regions, and merges tokens within flat areas toward low-energy destinations with explicit token recovery. We further provide a spectral graph coarsening view showing that score-guided merging yields bounded Laplacian spectral distortion compared to random or window-restricted baselines. Across eight natural and medical benchmarks, StructSAM reduces encoder FLOPs by 25-30% (up to 40%+ with prompt-aware merging) with minor drops in mIoU/Dice, consistently outperforming ToMe, PiToMe, ToMeSD, VidToMe, and ALGM at the same compute.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 45 of its 59 references that are in the map. First posted on arXiv 7 Mar 2026.

Builds on

45 of 59 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edge1 Jul
  2. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers14 Jun
  3. 2024Accelerating Transformers with Spectrum-Preserving Token Merging25 May
  4. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging2 Dec
  5. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation2 Aug
  6. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformers29 May
  7. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers27 May
  8. 2022Token Merging: Your ViT But Faster17 Oct

Other references · 37

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Segment Anythingcited by 8,158
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  9. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  10. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 345
  11. 2023Fast Segment Anythingcited by 286
  12. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
25 more
  1. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  2. 2023Token Merging for Fast Stable Diffusioncited by 220
  3. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  4. 2023VidToMe: Video Token Merging for Zero-Shot Video Editingcited by 111
  5. 2022Highly Accurate Dichotomous Image Segmentationcited by 110
  6. 2019Towards High-Resolution Salient Object Detectioncited by 92
  7. 2018Spectrally approximating large graphs with smaller graphscited by 56
  8. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  9. 2025ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Modelscited by 43
  10. 2018Graph Coarsening with Preserved Spectral Propertiescited by 41
  11. 2023EdgeSAM: Prompt-In-the-Loop Distillation for SAMcited by 40
  12. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  13. 2024PTQ4SAM: Post-Training Quantization for Segment Anythingcited by 32
  14. 2023SAM Meets Robotic Surgery: An Empirical Study on Generalization, Robustness and Adaptationcited by 30
  15. 2023Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformerscited by 22
  16. 2023SlimSAM: 0.1% Data Makes Segment Anything Slimcited by 20
  17. 2024GraspSAM: When Segment Anything Model Meets Grasp Detectioncited by 12
  18. 2023Revisiting Token Dropping Strategy in Efficient BERT Pretrainingcited by 11
  19. 2023AiluRus: A Scalable ViT Framework for Dense Predictioncited by 10
  20. 2025Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Groundingcited by 10
  21. 2025SlotVLA: Towards Modeling of Object-Relation Representations in Robotic Manipulationcited by 9
  22. 2023Dynamic Token-Pass Transformers for Semantic Segmentationcited by 7
  23. 2025IA-VLA: Input Augmentation for Vision-Language-Action models in settings with semantically complex taskscited by 4
  24. 2025How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?cited by 4
  25. 2026GPU-Accelerated Algorithms for Graph Vector Search: Taxonomy, Empirical Study, and Research Directionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SparseSAM: Structured Sparsification of Activations in Segment Anything Modelscited by 1
  2. 2026Prompt-Guided Interactive Segmentation of Interstitial Lung Disease in Thoracic CTcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
  2. 2026Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM
  3. 2026MergeOver: Post-Training Token Merging for Recursive Vision Transformers
  4. 2023Training-Free Acceleration of ViTs with Delayed Spatial Merging
  5. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers
  6. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
  7. 2025Lossless Token Merging Even Without Fine-Tuning in Vision Transformers
  8. 2025Similarity-Aware Token Pruning: Your VLM but Faster