# Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference

arXiv: https://arxiv.org/abs/2407.12893 · HTML: https://mlsanity.com/paper/2407.12893/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [Blockwise Self-Attention for Long Document Understanding](https://mlsanity.com/paper/1911.02972/) (2019)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://mlsanity.com/paper/1908.08962/) (2019)
- [A3: Accelerating Attention Mechanisms in Neural Networks with Approximation](https://mlsanity.com/paper/2002.10941/) (2020)
- [Adaptive Quantization for Deep Neural Network](https://mlsanity.com/paper/1712.01048/) (2017)
- [Visualizing and Understanding the Effectiveness of BERT](https://mlsanity.com/paper/1908.05620/) (2019)
- [An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers](https://mlsanity.com/paper/2208.06118/) (2022)
- [Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attention](https://mlsanity.com/paper/2110.09310/) (2021)
- [AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers](https://mlsanity.com/paper/2302.14705/) (2023)
- [Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads](https://mlsanity.com/paper/2011.03770/) (2020)
- [On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers](https://mlsanity.com/paper/2106.01335/) (2021)
- [P-Transformer: Towards Better Document-to-Document Neural Machine Translation](https://mlsanity.com/paper/2212.05830/) (2022)

## Built on by (most cited first, 2 of 2)

- [Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning](https://mlsanity.com/paper/2510.13832/) (2025)
- [Adaptive Head Budgeting for Efficient Multi-Head Attention](https://mlsanity.com/paper/2604.22583/) (2026)

## Abstract (arXiv)

In the world of deep learning, Transformer models have become very significant, leading to improvements in many areas from understanding language to recognizing images, covering a wide range of applications. Despite their success, the deployment of these models in real-time applications, particularly on edge devices, poses significant challenges due to their quadratic computational intensity and memory demands. To overcome these challenges we introduce a novel Hybrid Dynamic Pruning (HDP), an efficient algorithm-architecture co-design approach that accelerates transformers using head sparsity, block sparsity and approximation opportunities to reduce computations in attention and reduce memory access. With the observation of the huge redundancy in attention scores and attention heads, we propose a novel integer-based row-balanced block pruning to prune unimportant blocks in the attention matrix at run time, also propose integer-based head pruning to detect and prune unimportant heads at an early stage at run time. Also we propose an approximation method that reduces attention computations. To efficiently support these methods with lower latency and power efficiency, we propose a HDP co-processor architecture.