# Performance Evaluation of Swin Vision Transformer Model using Gradient Accumulation Optimization Technique

arXiv: https://arxiv.org/abs/2308.00197 · HTML: https://mlsanity.com/paper/2308.00197/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 8 of 8)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [A Survey of the Recent Architectures of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1901.06032/) (2019)
- [A Survey of Visual Transformers](https://mlsanity.com/paper/2111.06091/) (2021)
- [ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond](https://mlsanity.com/paper/2202.10108/) (2022)
- [TransNorm: Transformer Provides a Strong Spatial Normalization Mechanism for a Deep Segmentation Model](https://mlsanity.com/paper/2207.13415/) (2022)
- [Accumulated Gradient Normalization](https://mlsanity.com/paper/1710.02368/) (2017)

## Built on by (most cited first, 3 of 3)

- [Parameter-Selective Continual Test-Time Adaptation](https://mlsanity.com/paper/2407.02253/) (2024)
- [Quantifying Overfitting: Introducing the Overfitting Index](https://mlsanity.com/paper/2308.08682/) (2023)
- [An Ensemble Approach to Question Classification: Integrating Electra Transformer, GloVe, and LSTM](https://mlsanity.com/paper/2308.06828/) (2023)

## Abstract (arXiv)

Vision Transformers (ViTs) have emerged as a promising approach for visual recognition tasks, revolutionizing the field by leveraging the power of transformer-based architectures. Among the various ViT models, Swin Transformers have gained considerable attention due to their hierarchical design and ability to capture both local and global visual features effectively. This paper evaluates the performance of Swin ViT model using gradient accumulation optimization (GAO) technique. We investigate the impact of gradient accumulation optimization technique on the model's accuracy and training time. Our experiments show that applying the GAO technique leads to a significant decrease in the accuracy of the Swin ViT model, compared to the standard Swin Transformer model. Moreover, we detect a significant increase in the training time of the Swin ViT model when GAO model is applied. These findings suggest that applying the GAO technique may not be suitable for the Swin ViT model, and concern should be undertaken when using GAO technique for other transformer-based models.