# Life Regression based Patch Slimming for Vision Transformers

arXiv: https://arxiv.org/abs/2304.04926 · HTML: https://mlsanity.com/paper/2304.04926/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer](https://mlsanity.com/paper/2108.01390/) (2021)
- [All Tokens Matter: Token Labeling for Training Better Vision Transformers](https://mlsanity.com/paper/2104.10858/) (2021)
- [IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers](https://mlsanity.com/paper/2106.12620/) (2021)
- [Vision Transformer Pruning](https://mlsanity.com/paper/2104.08500/) (2021)
- [Learning Disentangled Representation Implicitly via Transformer for Occluded Person Re-Identification](https://mlsanity.com/paper/2107.02380/) (2021)
- [Language Models are Few-shot Learners for Prognostic Prediction](https://mlsanity.com/paper/2302.12692/) (2023)
- [DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition](https://mlsanity.com/paper/2302.01791/) (2023)
- [Real-time 3D Single Object Tracking with Transformer](https://mlsanity.com/paper/2209.00860/) (2022)
- [Single-Layer Vision Transformers for More Accurate Early Exits with Less Overhead](https://mlsanity.com/paper/2105.09121/) (2021)
- [LightSeq2: Accelerated Training for Transformer-based Models on GPUs](https://mlsanity.com/paper/2110.05722/) (2021)

## Built on by (most cited first, 1 of 1)

- [ClothPPO: A Proximal Policy Optimization Enhancing Framework for Robotic Cloth Manipulation with Observation-Aligned Action Spaces](https://mlsanity.com/paper/2405.04549/) (2024)

## Abstract (arXiv)

Vision transformers have achieved remarkable success in computer vision tasks by using multi-head self-attention modules to capture long-range dependencies within images. However, the high inference computation cost poses a new challenge. Several methods have been proposed to address this problem, mainly by slimming patches. In the inference stage, these methods classify patches into two classes, one to keep and the other to discard in multiple layers. This approach results in additional computation at every layer where patches are discarded, which hinders inference acceleration. In this study, we tackle the patch slimming problem from a different perspective by proposing a life regression module that determines the lifespan of each image patch in one go. During inference, the patch is discarded once the current layer index exceeds its life. Our proposed method avoids additional computation and parameters in multiple layers to enhance inference speed while maintaining competitive performance. Additionally, our approach requires fewer training epochs than other patch slimming methods.