# TAP-ViTs: Task-Adaptive Pruning for On-Device Deployment of Vision Transformers

arXiv: https://arxiv.org/abs/2601.02437 · HTML: https://mlsanity.com/paper/2601.02437/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Builds on (most cited first, 27 of 27)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [Dynamic Model Pruning with Feedback](https://mlsanity.com/paper/2006.07253/) (2020)
- [EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications](https://mlsanity.com/paper/2206.10589/) (2022)
- [Vision Transformer Pruning](https://mlsanity.com/paper/2104.08500/) (2021)
- [Global Vision Transformer Pruning with Hessian-Aware Saliency](https://mlsanity.com/paper/2110.04869/) (2021)
- [X-Pruner: eXplainable Pruning for Vision Transformers](https://mlsanity.com/paper/2303.04935/) (2023)
- [Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers](https://mlsanity.com/paper/2305.17328/) (2023)
- [CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction](https://mlsanity.com/paper/2203.04570/) (2022)
- [LPViT: Low-Power Semi-structured Pruning for Vision Transformers](https://mlsanity.com/paper/2407.02068/) (2024)
- [Dense Vision Transformer Compression with Few Samples](https://mlsanity.com/paper/2403.18708/) (2024)
- [DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices](https://mlsanity.com/paper/2412.05823/) (2024)
- [Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression](https://mlsanity.com/paper/2403.15835/) (2024)
- [Accelerating ViT Inference on FPGA through Static and Dynamic Pruning](https://mlsanity.com/paper/2403.14047/) (2024)
- [AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption](https://mlsanity.com/paper/2505.24773/) (2025)
- [Reinforcement Learning-based Token Pruning in Vision Transformers: A Markov Game Approach](https://mlsanity.com/paper/2503.23459/) (2025)
- [Research on Personalized Compression Algorithm for Pre-trained Models Based on Homomorphic Entropy Increase](https://mlsanity.com/paper/2408.08684/) (2024)

## Built on by (most cited first, 1 of 1)

- [Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models](https://mlsanity.com/paper/2607.13500/) (2026)

## Abstract (arXiv)

Vision Transformers (ViTs) have demonstrated strong performance across a wide range of vision tasks, yet their substantial computational and memory demands hinder efficient deployment on resource-constrained mobile and edge devices. Pruning has emerged as a promising direction for reducing ViT complexity. However, existing approaches either (i) produce a single pruned model shared across all devices, ignoring device heterogeneity, or (ii) rely on fine-tuning with device-local data, which is often infeasible due to limited on-device resources and strict privacy constraints. As a result, current methods fall short of enabling task-customized ViT pruning in privacy-preserving mobile computing settings. This paper introduces TAP-ViTs, a novel task-adaptive pruning framework that generates device-specific pruned ViT models without requiring access to any raw local data. Specifically, to infer device-level task characteristics under privacy constraints, we propose a Gaussian Mixture Model (GMM)-based metric dataset construction mechanism. Each device fits a lightweight GMM to approximate its private data distribution and uploads only the GMM parameters. Using these parameters, the cloud selects distribution-consistent samples from public data to construct a task-representative metric dataset for each device. Based on this proxy dataset, we further develop a dual-granularity importance evaluation-based pruning strategy that jointly measures composite neuron importance and adaptive layer importance, enabling fine-grained, task-aware pruning tailored to each device's computational budget. Extensive experiments across multiple ViT backbones and datasets demonstrate that TAP-ViTs consistently outperforms state-of-the-art pruning methods under comparable compression ratios.