# AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers

arXiv: https://arxiv.org/abs/2302.14705 · HTML: https://mlsanity.com/paper/2302.14705/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 72 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://mlsanity.com/paper/1908.08962/) (2019)
- [A3: Accelerating Attention Mechanisms in Neural Networks with Approximation](https://mlsanity.com/paper/2002.10941/) (2020)
- [Visualizing and Understanding the Effectiveness of BERT](https://mlsanity.com/paper/1908.05620/) (2019)
- [FTRANS: Energy-Efficient Acceleration of Transformers using FPGA](https://mlsanity.com/paper/2007.08563/) (2020)
- [Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer](https://mlsanity.com/paper/2009.08605/) (2020)
- [FastFormers: Highly Efficient Transformer Models for Natural Language Understanding](https://mlsanity.com/paper/2010.13382/) (2020)
- [Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attention](https://mlsanity.com/paper/2110.09310/) (2021)
- [NAAS: Neural Accelerator Architecture Search](https://mlsanity.com/paper/2105.13258/) (2021)
- [FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?](https://mlsanity.com/paper/2205.11656/) (2022)

## Built on by (most cited first, 24 of 24)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [SparseLLM: Towards Global Pruning for Pre-trained Language Models](https://mlsanity.com/paper/2402.17946/) (2024)
- [Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion](https://mlsanity.com/paper/2310.04361/) (2023)
- [QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edge](https://mlsanity.com/paper/2503.16709/) (2025)
- [HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline](https://mlsanity.com/paper/2407.17879/) (2024)
- [DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token Sampling](https://mlsanity.com/paper/2405.00888/) (2024)
- [MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness](https://mlsanity.com/paper/2509.10372/) (2025)
- [The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving](https://mlsanity.com/paper/2405.11299/) (2024)
- [Hardware Acceleration of LLMs: A comprehensive survey and comparison](https://mlsanity.com/paper/2409.03384/) (2024)
- [FedSpaLLM: Federated Pruning of Large Language Models](https://mlsanity.com/paper/2410.14852/) (2024)
- [EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms](https://mlsanity.com/paper/2303.13745/) (2023)
- [Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference](https://mlsanity.com/paper/2407.12893/) (2024)
- [Atleus: Accelerating Transformers on the Edge Enabled by 3D Heterogeneous Manycore Architectures](https://mlsanity.com/paper/2501.09588/) (2025)
- [Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration](https://mlsanity.com/paper/2505.17787/) (2025)
- [TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference](https://mlsanity.com/paper/2303.14882/) (2023)
- [Optimizing Foundation Model Inference on a Many-tiny-core Open-source RISC-V Platform](https://mlsanity.com/paper/2405.19284/) (2024)
- [A Low-Power Streaming Speech Enhancement Accelerator For Edge Devices](https://mlsanity.com/paper/2503.21335/) (2025)
- [SSVQ: Unleashing the Potential of Vector Quantization with Sign-Splitting](https://mlsanity.com/paper/2503.08668/) (2025)
- [Bhasha-Rupantarika: Algorithm-Hardware Co-design approach for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2510.10676/) (2025)
- [QUILL: An Algorithm-Architecture Co-Design for Cache-Local Deformable Attention](https://mlsanity.com/paper/2511.13679/) (2025)
- [Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding](https://mlsanity.com/paper/2604.24820/) (2026)
- [ThRIve: Thermally Robust CNN Inference via Low-Rank Adaptation in Heterogeneous PIM Architectures](https://mlsanity.com/paper/2607.17091/) (2026)
- [DeVIT: Low-Power Vision Transformer Acceleration Using Delta Computation](https://mlsanity.com/paper/2608.01343/) (2026)
- [HYDRA: A Heterogeneous Chiplet DSE Framework for Serving Dynamic Hybrid LLM Workloads](https://mlsanity.com/paper/2608.19395/) (2026)

## Abstract (arXiv)

Self-attention-based transformer models have achieved tremendous success in the domain of natural language processing. Despite their efficacy, accelerating the transformer is challenging due to its quadratic computational complexity and large activation sizes. Existing transformer accelerators attempt to prune its tokens to reduce memory access, albeit with high compute overheads. Moreover, previous works directly operate on large matrices involved in the attention operation, which limits hardware utilization. In order to address these challenges, this work proposes a novel dynamic inference scheme, DynaTran, which prunes activations at runtime with low overhead, substantially reducing the number of ineffectual operations. This improves the throughput of transformer inference. We further propose tiling the matrices in transformer operations along with diverse dataflows to improve data reuse, thus enabling higher energy efficiency. To effectively implement these methods, we propose AccelTran, a novel accelerator architecture for transformers. Extensive experiments with different models and benchmarks demonstrate that DynaTran achieves higher accuracy than the state-of-the-art top-k hardware-aware pruning strategy while attaining up to 1.2× higher sparsity. One of our proposed accelerators, AccelTran-Edge, achieves 330K× higher throughput with 93K× lower energy requirement when compared to a Raspberry Pi device. On the other hand, AccelTran-Server achieves 5.73× higher throughput and 3.69× lower energy consumption compared to the state-of-the-art transformer co-processor, Energon. The simulation source code is available at https://github.com/jha-lab/acceltran.