# Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attention

arXiv: https://arxiv.org/abs/2110.09310 · HTML: https://mlsanity.com/paper/2110.09310/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 86 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)

## Built on by (most cited first, 26 of 26)

- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design](https://mlsanity.com/paper/2209.09570/) (2022)
- [PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformation](https://mlsanity.com/paper/2301.10936/) (2023)
- [AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers](https://mlsanity.com/paper/2302.14705/) (2023)
- [A Survey on Hardware Accelerators for Large Language Models](https://mlsanity.com/paper/2401.09890/) (2024)
- [ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer](https://mlsanity.com/paper/2306.06446/) (2023)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [Identifying Unnecessary 3D Gaussians using Clustering for Fast Rendering of 3D Gaussian Splatting](https://mlsanity.com/paper/2402.13827/) (2024)
- [Sparse-DySta: Sparsity-Aware Dynamic and Static Scheduling for Sparse Multi-DNN Workloads](https://mlsanity.com/paper/2310.11096/) (2023)
- [Deploying Foundation Model Powered Agent Services: A Survey](https://mlsanity.com/paper/2412.13437/) (2024)
- [MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness](https://mlsanity.com/paper/2509.10372/) (2025)
- [The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving](https://mlsanity.com/paper/2405.11299/) (2024)
- [Hardware Acceleration of LLMs: A comprehensive survey and comparison](https://mlsanity.com/paper/2409.03384/) (2024)
- [A Practical Tensor-Network Compression Pipeline for Production-Scale Large Language Models](https://mlsanity.com/paper/2602.01613/) (2026)
- [EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms](https://mlsanity.com/paper/2303.13745/) (2023)
- [LATTE: Low-Precision Approximate Attention with Head-wise Trainable Threshold for Efficient Transformer](https://mlsanity.com/paper/2404.07519/) (2024)
- [Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference](https://mlsanity.com/paper/2407.12893/) (2024)
- [Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding](https://mlsanity.com/paper/2502.08363/) (2025)
- [GS-TG: 3D Gaussian Splatting Accelerator with Tile Grouping for Reducing Redundant Sorting while Preserving Rasterization Efficiency](https://mlsanity.com/paper/2509.00911/) (2025)
- [Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity](https://mlsanity.com/paper/2510.15964/) (2025)
- [TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference](https://mlsanity.com/paper/2303.14882/) (2023)
- [Energy-Efficient Wireless LLM Inference via Uncertainty and Importance-Aware Speculative Decoding](https://mlsanity.com/paper/2508.12590/) (2025)
- [BitStopper: An Efficient Transformer Attention Accelerator via Stage-fusion and Early Termination](https://mlsanity.com/paper/2512.06457/) (2025)
- [LAPA: Log-Domain Prediction-Driven Dynamic Sparsity Accelerator for Transformer Model](https://mlsanity.com/paper/2512.07855/) (2025)
- [Chimera: Neuro-Symbolic Attention Primitives for Trustworthy Dataplane Intelligence](https://mlsanity.com/paper/2602.12851/) (2026)
- [Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding](https://mlsanity.com/paper/2604.24820/) (2026)

## Abstract (arXiv)

In recent years, transformer models have revolutionized Natural Language Processing (NLP) and shown promising performance on Computer Vision (CV) tasks. Despite their effectiveness, transformers' attention operations are hard to accelerate due to the complicated data movement and quadratic computational complexity, prohibiting the real-time inference on resource-constrained edge-computing platforms. To tackle this challenge, we propose Energon, an algorithm-architecture co-design approach that accelerates various transformers using dynamic sparse attention. With the observation that attention results only depend on a few important query-key pairs, we propose a Mix-Precision Multi-Round Filtering (MP-MRF) algorithm to dynamically identify such pairs at runtime. We adopt low bitwidth in each filtering round and only use high-precision tensors in the attention stage to reduce overall complexity. By this means, we significantly mitigate the computational cost with negligible accuracy loss. To enable such an algorithm with lower latency and better energy efficiency, we also propose an Energon co-processor architecture. Elaborated pipelines and specialized optimizations jointly boost the performance and reduce power consumption. Extensive experiments on both NLP and CV benchmarks demonstrate that Energon achieves 168× and 8.7× geo-mean speedup and up to 104× and 103× energy reduction compared with Intel Xeon 5220 CPU and NVIDIA V100 GPU. Compared to state-of-the-art attention accelerators SpAtten and A3, Energon also achieves 1.7×, 1.25× speedup and 1.6 ×, 1.5×  higher energy efficiency.