mlsanityData to 7 Oct 2026

ClustViT: Clustering-based Token Merging for Semantic Segmentation

arXiv 2510.01948 ↗First posted 2 Oct 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Vision Transformers can achieve high accuracy and strong generalization across various contexts, but their practical applicability on real-world robotic systems is limited due to their quadratic attention complexity. Recent works have focused on dynamically merging tokens according to the image complexity. Token merging works well for classification but is less suited to dense prediction. We propose ClustViT, where we expand upon the Vision Transformer (ViT) backbone and address semantic segmentation. Within our architecture, a trainable Cluster module merges similar tokens along the network guided by pseudo-clusters from segmentation masks. Subsequently, a Regenerator module restores fine details for downstream heads. Our approach achieves up to 2.18x fewer GFLOPs and 1.64x faster inference on three different datasets, with comparable segmentation accuracy. Our code and models will be made publicly available.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

21 of 27 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  7. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  8. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  9. 2022Token Merging: Your ViT But Fastercited by 995
  10. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  11. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  12. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
9 more
  1. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  2. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 105
  3. 2020Semantic Segmentation of Underwater Imagery: Dataset and Benchmarkcited by 95
  4. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentationcited by 61
  5. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  6. 2023Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformerscited by 22
  7. 2023GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagationcited by 19
  8. 2022SaiT: Sparse Vision Transformers through Adaptive Token Pruningcited by 15
  9. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM
  2. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
  3. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers
  4. 2024Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens
  5. 2024Segformer++: Efficient Token-Merging Strategies for High-Resolution Semantic Segmentation
  6. 2024Token Cropr: Faster ViTs for Quite a Few Tasks
  7. 2023MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers
  8. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging