mlsanityData to 14 Oct 2026

SparseSwin: Swin Transformer with Sparse Transformer Block

arXiv 2309.05224 ↗First posted 11 Sep 2023cs.CV cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Advancements in computer vision research have put transformer architecture as the state of the art in computer vision tasks. One of the known drawbacks of the transformer architecture is the high number of parameters, this can lead to a more complex and inefficient algorithm. This paper aims to reduce the number of parameters and in turn, made the transformer more efficient. We present Sparse Transformer (SparTa) Block, a modified transformer block with an addition of a sparse token converter that reduces the number of tokens used. We use the SparTa Block inside the Swin T architecture (SparseSwin) to leverage Swin capability to downsample its input and reduce the number of initial tokens to be calculated. The proposed SparseSwin model outperforms other state of the art models in image classification with an accuracy of 86.96%, 97.43%, and 85.35% on the ImageNet100, CIFAR10, and CIFAR100 datasets respectively. Despite its fewer parameters, the result highlights the potential of a transformer architecture using a sparse token converter with a limited number of tokens to optimize the use of the transformer and improve its performance.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 35 of its 62 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

35 of 62 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  10. 2016Densely Connected Convolutional Networkscited by 6,720
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  12. 2016Layer Normalizationcited by 5,669
23 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  3. 2014Going Deeper with Convolutionscited by 4,241
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  5. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  7. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  9. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  10. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  11. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
  12. 2014Recurrent Models of Visual Attentioncited by 349
  13. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  14. 2018Neural Baby Talkcited by 107
  15. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  16. 2023White-Box Transformers via Sparse Rate Reductioncited by 80
  17. 2017Oriented Response Networkscited by 68
  18. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  19. 2023Vision Transformer with Quadrangle Attentioncited by 46
  20. 2022DLME: Deep Local-flatness Manifold Embeddingcited by 16
  21. 2023Hybrid Transformer and CNN Attention Network for Stereo Image Super-resolutioncited by 10
  22. 2023SparseFormer: Sparse Visual Recognition via Limited Latent Tokenscited by 9
  23. 2021XnODR and XnIDR: Two Accurate and Fast Fully Connected Layers For Convolutional Neural Networkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2026A neurosymbolic Approach with Epistemic Deep Learning for Hierarchical Image Classificationcited by 1
  2. 2024Robust Network Learning via Inverse Scale Variational Sparsificationcited by 0
  3. 2025Escaping The Big Data Paradigm in Self-Supervised Representation Learningcited by 0
  4. 2025Spatial-Temporal-Spectral Mamba with Sparse Deformable Token Sequence for Enhanced MODIS Time Series Classificationcited by 0