# SimViT: Exploring a Simple Vision Transformer with sliding windows

arXiv: https://arxiv.org/abs/2112.13085 · HTML: https://mlsanity.com/paper/2112.13085/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 24 Dec 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Local Relation Networks for Image Recognition](https://mlsanity.com/paper/1904.11491/) (2019)

## Built on by (most cited first, 8 of 8)

- [Stepwise Feature Fusion: Local Guides Global](https://mlsanity.com/paper/2203.03635/) (2022)
- [LightViT: Towards Light-Weight Convolution-Free Vision Transformers](https://mlsanity.com/paper/2207.05557/) (2022)
- [DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition](https://mlsanity.com/paper/2302.01791/) (2023)
- [Do we really need temporal convolutions in action segmentation?](https://mlsanity.com/paper/2205.13425/) (2022)
- [Factorization Vision Transformer: Modeling Long Range Dependency with Local Window Cost](https://mlsanity.com/paper/2312.08614/) (2023)
- [Deformable Attention Graph Representation Learning for Histopathology Whole Slide Image Analysis](https://mlsanity.com/paper/2508.05382/) (2025)
- [DualToken-ViT: Position-aware Efficient Vision Transformer with Dual Token Fusion](https://mlsanity.com/paper/2309.12424/) (2023)
- [Hilbert-Guided Sparse Local Attention](https://mlsanity.com/paper/2511.05832/) (2025)

## Abstract (arXiv)

Although vision Transformers have achieved excellent performance as backbone models in many vision tasks, most of them intend to capture global relations of all tokens in an image or a window, which disrupts the inherent spatial and local correlations between patches in 2D structure. In this paper, we introduce a simple vision Transformer named SimViT, to incorporate spatial structure and local information into the vision Transformers. Specifically, we introduce Multi-head Central Self-Attention(MCSA) instead of conventional Multi-head Self-Attention to capture highly local relations. The introduction of sliding windows facilitates the capture of spatial structure. Meanwhile, SimViT extracts multi-scale hierarchical features from different layers for dense prediction tasks. Extensive experiments show the SimViT is effective and efficient as a general-purpose backbone model for various image processing tasks. Especially, our SimViT-Micro only needs 3.3M parameters to achieve 71.1% top-1 accuracy on ImageNet-1k dataset, which is the smallest size vision Transformer model by now. Our code will be available in https://github.com/ucasligang/SimViT.