mlsanityData to 7 Oct 2026

Learned Token Pruning for Transformers

arXiv 2107.00910 ↗First posted 2 Jul 2021cs.CLcited by 138

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Deploying transformer models in practice is challenging due to their inference cost, which scales quadratically with input sequence length. To address this, we present a novel Learned Token Pruning (LTP) method which adaptively removes unimportant tokens as an input sequence passes through transformer layers. In particular, LTP prunes tokens with an attention score below a threshold value which is learned for each layer during training. Our threshold-based method allows the length of the pruned sequence to vary adaptively based on the input sequence, and avoids algorithmically expensive operations such as top-k token selection. We extensively test the performance of LTP on GLUE tasks and show that our method outperforms the prior state-of-the-art token pruning methods by up to ~2.5% higher accuracy with the same amount of FLOPs. In particular, LTP achieves up to 2.1x FLOPs reduction with less than 1% accuracy drop, which results in up to 1.9x and 2.0x throughput improvement on Intel Haswell CPUs and NVIDIA V100 GPUs, respectively. Furthermore, we demonstrate that LTP is more robust than prior methods to variations on input sentence lengths. Our code has been developed in PyTorch and has been open-sourced.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 63 of its 80 references that are in the map. First posted on arXiv 2 Jul 2021.

Builds on

63 of 80 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2020Reformer: The Efficient Transformercited by 1,658
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2019Are Sixteen Heads Really Better than One?cited by 803
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  11. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  12. 2019Learned Step Size Quantizationcited by 604
  13. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  14. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  15. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  16. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  17. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  18. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  19. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  20. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  21. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  22. 2020Sparse Sinkhorn Attentioncited by 250
  23. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  24. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  25. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  26. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  27. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  28. 2019Structured Pruning of Large Language Modelscited by 201
  29. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  30. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  31. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  32. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  33. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  34. 2020Fast Transformers with Clustered Attentioncited by 123
  35. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  36. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  37. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  38. 2020Bayesian Bits: Unifying Quantization and Pruningcited by 93

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 15; 2023: 34; 2024: 36; 2025: 35; 2026: 14.

138 papers.

Sort:

By area:

  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  3. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  4. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  5. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  6. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  7. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  8. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  9. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  10. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  11. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  12. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
38 more
  1. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  3. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  4. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  5. 2024Fast On-device LLM Inference with NPUscited by 58
  6. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  7. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  8. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  9. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  10. 2024A Survey on Transformer Compressioncited by 44
  11. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  12. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  13. 2024Scene Adaptive Sparse Transformer for Event-based Object Detectioncited by 39
  14. 2023Less is More: Focus Attention for Efficient DETRcited by 38
  15. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  16. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attentioncited by 35
  17. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  18. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  19. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  20. 2023Towards Unified Deep Image Deraining: A Survey and A New Benchmarkcited by 34
  21. 2025The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruningcited by 34
  22. 2023DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Modelscited by 30
  23. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  24. 2025Cognitive Memory in Large Language Modelscited by 28
  25. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  26. 2023Faster Causal Attention Over Large Sequences Through Sparse Flash Attentioncited by 24
  27. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  28. 2023Learned Thresholds Token Merging and Pruning for Vision Transformerscited by 22
  29. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understandingcited by 22
  30. 2022Transkimmer: Transformer Learns to Layer-wise Skimcited by 21
  31. 2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 21
  32. 2022CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Modelscited by 20
  33. 2024SpeechPrune: Context-aware Token Pruning for Speech Information Retrievalcited by 20
  34. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Modelscited by 19
  35. 2024Characterizing Prompt Compression Methods for Long Context Inferencecited by 18
  36. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 18
  37. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 16
  38. 2021Magic Pyramid: Accelerating Inference with Early Exiting and Token Pruningcited by 15

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Token Compression for Vision Transformer with Spatial Information Preserved
  2. 2026Adaptive Computation Depth via Learned Token Routing in Transformers
  3. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  4. 2026TP-Spikformer: Token Pruned Spiking Transformer
  5. 2024Exploring Token Pruning in Vision State Space Models
  6. 2025TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training
  7. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning
  8. 2026Mixture of Chapters: Scaling Learnt Memory in Transformers