# SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection

arXiv: https://arxiv.org/abs/2401.15293 · HTML: https://mlsanity.com/paper/2401.15293/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 27 Jan 2024.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [A Short Study on Compressing Decoder-Based Language Models](https://mlsanity.com/paper/2110.08460/) (2021)
- [GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values](https://mlsanity.com/paper/2311.03426/) (2023)
- [Gray-box Adversarial Attack of Deep Reinforcement Learning-based Trading Agents](https://mlsanity.com/paper/2309.14615/) (2023)

## Built on by (most cited first, 5 of 5)

- [Accelerating the Low-Rank Decomposed Models](https://mlsanity.com/paper/2407.20266/) (2024)
- [Unlocking Full Efficiency of Token Filtering in Large Language Model Training](https://mlsanity.com/paper/2502.00340/) (2025)
- [Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?](https://mlsanity.com/paper/2407.16514/) (2024)
- [EPAS: Efficient Training with Progressive Activation Sharing](https://mlsanity.com/paper/2601.19089/) (2026)
- [APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention](https://mlsanity.com/paper/2601.21444/) (2026)

## Abstract (arXiv)

Vision transformers are known to be more computationally and data-intensive than CNN models. These transformer models such as ViT, require all the input image tokens to learn the relationship among them. However, many of these tokens are not informative and may contain irrelevant information such as unrelated background or unimportant scenery. These tokens are overlooked by the multi-head self-attention (MHSA), resulting in many redundant and unnecessary computations in MHSA and the feed-forward network (FFN). In this work, we propose a method to optimize the amount of unnecessary interactions between unimportant tokens by separating and sending them through a different low-cost computational path. Our method does not add any parameters to the ViT model and aims to find the best trade-off between training throughput and achieving a 0% loss in the Top-1 accuracy of the final model. Our experimental results on training ViT-small from scratch show that SkipViT is capable of effectively dropping 55% of the tokens while gaining more than 13% training throughput and maintaining classification accuracy at the level of the baseline model on Huawei Ascend910A.