# Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training

arXiv: https://arxiv.org/abs/2211.10801 · HTML: https://mlsanity.com/paper/2211.10801/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 63 references that are in the map.
- First posted on arXiv 19 Nov 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Not All Samples Are Created Equal: Deep Learning with Importance Sampling](https://mlsanity.com/paper/1803.00942/) (2018)

## Built on by (most cited first, 24 of 24)

- [Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality](https://mlsanity.com/paper/2505.18227/) (2025)
- [LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers](https://mlsanity.com/paper/2412.12444/) (2024)
- [Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge](https://mlsanity.com/paper/2312.05693/) (2023)
- [Exploring Token Pruning in Vision State Space Models](https://mlsanity.com/paper/2409.18962/) (2024)
- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [Numerical Pruning for Efficient Autoregressive Models](https://mlsanity.com/paper/2412.12441/) (2024)
- [GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation](https://mlsanity.com/paper/2311.03035/) (2023)
- [DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Network](https://mlsanity.com/paper/2303.02165/) (2023)
- [DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing](https://mlsanity.com/paper/2212.03597/) (2022)
- [Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration](https://mlsanity.com/paper/2506.05709/) (2025)
- [RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers](https://mlsanity.com/paper/2505.21847/) (2025)
- [Open-Source Multimodal Moxin Models with Moxin-VLM and Moxin-VLA](https://mlsanity.com/paper/2512.22208/) (2025)
- [Reinforcement Learning-based Token Pruning in Vision Transformers: A Markov Game Approach](https://mlsanity.com/paper/2503.23459/) (2025)
- [MUSE: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonality](https://mlsanity.com/paper/2605.05646/) (2026)
- [Temporal Aware Pruning for Efficient Diffusion-based Video Generation](https://mlsanity.com/paper/2605.17837/) (2026)
- [Prototype-Driven Multi-Feature Generation for Visible-Infrared Person Re-identification](https://mlsanity.com/paper/2409.05642/) (2024)
- [EP-SAM: Weakly Supervised Histopathology Segmentation via Enhanced Prompt with Segment Anything](https://mlsanity.com/paper/2410.13621/) (2024)
- [CAFE-AD: Cross-Scenario Adaptive Feature Enhancement for Trajectory Planning in Autonomous Driving](https://mlsanity.com/paper/2504.06584/) (2025)
- [TSLA: A Task-Specific Learning Adaptation for Semantic Segmentation on Autonomous Vehicles Platform](https://mlsanity.com/paper/2508.12279/) (2025)
- [Hierarchical Indexing with Knowledge Enrichment for Multilingual Video Corpus Retrieval](https://mlsanity.com/paper/2510.09553/) (2025)
- [Distilling Token-Pruned Pose Transformer for 2D Human Pose Estimation](https://mlsanity.com/paper/2304.05548/) (2023)
- [Improved TokenPose with Sparsity](https://mlsanity.com/paper/2311.09653/) (2023)
- [TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditching](https://mlsanity.com/paper/2601.19739/) (2026)
- [HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers](https://mlsanity.com/paper/2603.12222/) (2026)

## Abstract (arXiv)

Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage at both training and inference time limit their generalization. Previous compression algorithms usually start from the pre-trained dense models and only focus on efficient inference, while time-consuming training is still unavoidable. In contrast, this paper points out that the million-scale training data is redundant, which is the fundamental reason for the tedious training. To address the issue, this paper aims to introduce sparsity into data and proposes an end-to-end efficient training framework from three sparse perspectives, dubbed Tri-Level E-ViT. Specifically, we leverage a hierarchical data redundancy reduction scheme, by exploring the sparsity under three levels: number of training examples in the dataset, number of patches (tokens) in each example, and number of connections between tokens that lie in attention weights. With extensive experiments, we demonstrate that our proposed technique can noticeably accelerate training for various ViT architectures while maintaining accuracy. Remarkably, under certain ratios, we are able to improve the ViT accuracy rather than compromising it. For example, we can achieve 15.2% speedup with 72.6% (+0.4) Top-1 accuracy on Deit-T, and 15.7% speedup with 79.9% (+0.1) Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT.