mlsanityData to 14 Oct 2026

SAC-ViT: Semantic-Aware Clustering Vision Transformer with Early Exit

arXiv 2503.00060 ↗First posted 27 Feb 2025cs.CV cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

The Vision Transformer (ViT) excels in global modeling but faces deployment challenges on resource-constrained devices due to the quadratic computational complexity of its attention mechanism. To address this, we propose the Semantic-Aware Clustering Vision Transformer (SAC-ViT), a non-iterative approach to enhance ViT's computational efficiency. SAC-ViT operates in two stages: Early Exit (EE) and Semantic-Aware Clustering (SAC). In the EE stage, downsampled input images are processed to extract global semantic information and generate initial inference results. If these results do not meet the EE termination criteria, the information is clustered into target and non-target tokens. In the SAC stage, target tokens are mapped back to the original image, cropped, and embedded. These target tokens are then combined with reused non-target tokens from the EE stage, and the attention mechanism is applied within each cluster. This two-stage design, with end-to-end optimization, reduces spatial redundancy and enhances computational efficiency, significantly boosting overall ViT performance. Extensive experiments demonstrate the efficacy of SAC-ViT, reducing 62% of the FLOPs of DeiT and achieving 1.98 times throughput without compromising performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

37 of 45 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  9. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  11. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  12. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
25 more
  1. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  2. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  3. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  4. 2021Going deeper with Image Transformerscited by 764
  5. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  6. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  7. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  8. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  9. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  10. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 177
  11. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  12. 2020Attentional Feature Fusioncited by 115
  13. 2021Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsitycited by 98
  14. 2019AI Benchmark: All About Deep Learning on Smartphones in 2019cited by 77
  15. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  16. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  17. 2023SpectFormer: Frequency and Attention is what you need in a Vision Transformercited by 48
  18. 2023SG-Former: Self-guided Transformer with Evolving Token Reallocationcited by 32
  19. 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 29
  20. 2023Lightweight Vision Transformer with Bidirectional Interactioncited by 25
  21. 2022SaiT: Sparse Vision Transformers through Adaptive Token Pruningcited by 15
  22. 2022Dynamic Group Transformer: A General Vision Transformer Backbone with Dynamic Group Attentioncited by 14
  23. 2023Bi-ViT: Pushing the Limit of Vision Transformer Quantizationcited by 13
  24. 2024LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognitioncited by 13
  25. 2022Lightweight Vision Transformer with Cross Feature Attentioncited by 5

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Synthetic-to-Real Dehazing Method based on Domain Unificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens
  2. 2026ViT-5: Vision Transformers for The Mid-2020s
  3. 2024Optimizing Vision Transformers with Data-Free Knowledge Transfer
  4. 2026VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
  5. 2024ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition
  6. 2026Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM
  7. 2025ClustViT: Clustering-based Token Merging for Semantic Segmentation
  8. 2023MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers