mlsanityData to 14 Oct 2026

HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers

arXiv 2211.08110 ↗First posted 15 Nov 2022cs.AR cs.AI cs.CVcited by 46

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

While vision transformers (ViTs) have continuously achieved new milestones in the field of computer vision, their sophisticated network architectures with high computation and memory costs have impeded their deployment on resource-limited edge devices. In this paper, we propose a hardware-efficient image-adaptive token pruning framework called HeatViT for efficient yet accurate ViT acceleration on embedded FPGAs. By analyzing the inherent computational patterns in ViTs, we first design an effective attention-based multi-head token selector, which can be progressively inserted before transformer blocks to dynamically identify and consolidate the non-informative tokens from input images. Moreover, we implement the token selector on hardware by adding miniature control logic to heavily reuse existing hardware components built for the backbone ViT. To improve the hardware efficiency, we further employ 8-bit fixed-point quantization, and propose polynomial approximations with regularization effect on quantization error for the frequently used nonlinear functions in ViTs. Finally, we propose a latency-aware multi-stage training strategy to determine the transformer blocks for inserting token selectors and optimize the desired (average) pruning rates for inserted token selectors, in order to improve both the model accuracy and inference latency on hardware. Compared to existing ViT pruning studies, under the similar computation cost, HeatViT can achieve 0.7%~8.9% higher accuracy; while under the similar model accuracy, HeatViT can achieve more than 28.4%~65.3% computation reduction, for various widely used ViTs, including DeiT-T, DeiT-S, DeiT-B, LV-ViT-S, and LV-ViT-M, on the ImageNet dataset. Compared to the baseline hardware accelerator, our implementations of HeatViT on the Xilinx ZCU102 FPGA achieve 3.46×~4.89× speedup.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 51 of its 69 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

51 of 69 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec
  2. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug

Other references · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2017Mask R-CNNcited by 7,277
  8. 2017Squeeze-and-Excitation Networkscited by 4,789
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
37 more
  1. 2019Searching for MobileNetV3cited by 1,994
  2. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  4. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  5. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  6. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  7. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  8. 2020Pre-Trained Image Processing Transformercited by 893
  9. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  10. 2021Transformer in Transformercited by 633
  11. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  12. 2021Post-Training Quantization for Vision Transformercited by 278
  13. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  14. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  15. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  16. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 188
  17. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  18. 2019The Evolved Transformercited by 180
  19. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  20. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  21. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  22. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  23. 2021Instance Localization for Self-supervised Detection Pretrainingcited by 99
  24. 2022Unified Visual Transformer Compressioncited by 79
  25. 2020SoftSort: A Continuous Relaxation for the argsort Operatorcited by 66
  26. 2021BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Searchcited by 63
  27. 2020FTRANS: Energy-Efficient Acceleration of Transformers using FPGAcited by 54
  28. 2021GLiT: Neural Architecture Search for Global and Local Image Transformercited by 53
  29. 2018Deep Learning using Rectified Linear Units (ReLU)cited by 48
  30. 2021Searching the Search Space of Vision Transformercited by 41
  31. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  32. 2021Refiner: Refining Self-attention for Vision Transformerscited by 38
  33. 2019Fully Quantized Transformer for Machine Translationcited by 36
  34. 2019NAT: Neural Architecture Transformer for Accurate and Compact Architecturescited by 33
  35. 2021Dual-stream Network for Visual Recognitioncited by 33
  36. 2022Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantizationcited by 29
  37. 2020An Investigation on Different Underlying Quantization Schemes for Pre-trained Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 12; 2025: 16; 2026: 12.

46 papers.

Sort:

By area:

  1. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 126
  2. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 62
  3. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  4. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  5. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  6. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  7. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  8. 2024Exploring Token Pruning in Vision State Space Modelscited by 28
  9. 2025Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategiescited by 24
  10. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 20
  11. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 19
  12. 2024Squat: Quant Small Language Models on the Edgecited by 16
34 more
  1. 2023DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Networkcited by 14
  2. 2024HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipelinecited by 7
  3. 2025Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGAcited by 7
  4. 2024Accelerating ViT Inference on FPGA through Static and Dynamic Pruningcited by 5
  5. 2025D'ej`a Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reusecited by 5
  6. 2024P2-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformercited by 4
  7. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  8. 2025UbiMoE: A Ubiquitous Mixture-of-Experts Vision Transformer Accelerator With Hybrid Computation Pattern on FPGAcited by 4
  9. 2025LL-ViT: Edge Deployable Vision Transformers with Look Up Table Neuronscited by 3
  10. 2025Focus: A Streaming Concentration Architecture for Efficient Vision-Language Modelscited by 3
  11. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Modelscited by 3
  12. 2024TReX- Reusing Vision Transformer's Attention for Efficient Xbar-based Computingcited by 2
  13. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  14. 2025Low Power Vision Transformer Accelerator with Hardware-Aware Pruning and Optimized Dataflowcited by 2
  15. 2026TriCCOT: Tri-part Convolutional Conformal Transformer for Onboard Space Object Detectioncited by 2
  16. 2025Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruningcited by 1
  17. 2025FaRAccel: FPGA-Accelerated Defense Architecture for Efficient Bit-Flip Attack Resilience in Transformer Modelscited by 1
  18. 2026FB-CLIP: Fine-Grained Zero-Shot Anomaly Detection with Foreground-Background Disentanglementcited by 1
  19. 2026Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectorscited by 1
  20. 2023Treasure What You Have: Exploiting Similarity in Deep Neural Networks for Efficient Video Processingcited by 0
  21. 2023NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix Operations for Efficient Inferencecited by 0
  22. 2024LUM-ViT: Learnable Under-sampling Mask Vision Transformer for Bandwidth Limited Optical Signal Acquisitioncited by 0
  23. 2024RO-SVD: A Reconfigurable Hardware Copyright Protection Framework for AIGC Applicationscited by 0
  24. 2025T-REX: A 68-567 μs/token, 0.41-3.95 μJ/token Transformer Accelerator with Reduced External Memory Access and Enhanced Hardware Utilization in 16nm FinFETcited by 0
  25. 2025CoQMoE: Co-Designed Quantization and Computation Orchestration for Mixture-of-Experts Vision Transformer on FPGAcited by 0
  26. 2025Polymorph: Energy-Efficient Multi-Label Classification for Video Streams on Embedded Devicescited by 0
  27. 2026ToaSt: Token Channel Selection and Structured Pruning for Efficient ViTcited by 0
  28. 2026TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AIcited by 0
  29. 2026DxPTA: An Architecture Design Space Exploration with Optical Dataflow-guided Strategy for HW/SW Co-Design of Photonic Transformer Acceleratorscited by 0
  30. 2026FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformerscited by 0
  31. 2026DeVIT: Low-Power Vision Transformer Acceleration Using Delta Computationcited by 0
  32. 2026Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inferencecited by 0
  33. 2026Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TMcited by 0
  34. 2026MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  2. 2023No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling
  3. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
  4. 2022SaiT: Sparse Vision Transformers through Adaptive Token Pruning
  5. 2024M2-ViT: Accelerating Hybrid Vision Transformers with Two-Level Mixed Quantization
  6. 2024Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edge
  7. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  8. 2024Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers