# Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization

arXiv: https://arxiv.org/abs/2110.10030 · HTML: https://mlsanity.com/paper/2110.10030/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 19 Oct 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [When BERT Plays the Lottery, All Tickets Are Winning](https://mlsanity.com/paper/2005.00561/) (2020)
- [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://mlsanity.com/paper/1804.09849/) (2018)
- [A3: Accelerating Attention Mechanisms in Neural Networks with Approximation](https://mlsanity.com/paper/2002.10941/) (2020)
- [Reasoning about Entailment with Neural Attention](https://mlsanity.com/paper/1509.06664/) (2015)
- [FTRANS: Energy-Efficient Acceleration of Transformers using FPGA](https://mlsanity.com/paper/2007.08563/) (2020)
- [PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devices](https://mlsanity.com/paper/1909.05073/) (2019)
- [Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks](https://mlsanity.com/paper/2002.04116/) (2020)
- [Block-Sparse Recurrent Neural Networks](https://mlsanity.com/paper/1711.02782/) (2017)
- [Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning](https://mlsanity.com/paper/2009.08065/) (2020)
- [Accuracy vs. Efficiency: Achieving Both through FPGA-Implementation Aware Neural Architecture Search](https://mlsanity.com/paper/1901.11211/) (2019)
- [Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity](https://mlsanity.com/paper/2008.13006/) (2020)
- [Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference](https://mlsanity.com/paper/1907.08985/) (2019)
- [Accelerating Sparse Matrix-Matrix Multiplication with GPU Tensor Cores](https://mlsanity.com/paper/2009.14600/) (2020)
- [CSR5: An Efficient Storage Format for Cross-Platform Sparse Matrix-Vector Multiplication](https://mlsanity.com/paper/1503.05032/) (2015)
- [CSB-RNN: A Faster-than-Realtime RNN Acceleration Framework with Compressed Structured Blocks](https://mlsanity.com/paper/2005.05758/) (2020)

## Built on by (most cited first, 27 of 27)

- [Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference](https://mlsanity.com/paper/2312.15159/) (2023)
- [AutoReP: Automatic ReLU Replacement for Fast Private Network Inference](https://mlsanity.com/paper/2308.10134/) (2023)
- [LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference](https://mlsanity.com/paper/2309.14331/) (2023)
- [A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining](https://mlsanity.com/paper/2208.03646/) (2022)
- [Accel-GCN: High-Performance GPU Accelerator Design for Graph Convolution Networks](https://mlsanity.com/paper/2308.11825/) (2023)
- [Towards Sparsification of Graph Neural Networks](https://mlsanity.com/paper/2209.04766/) (2022)
- [Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-off](https://mlsanity.com/paper/2211.16667/) (2022)
- [An Automatic and Efficient BERT Pruning for Edge AI Systems](https://mlsanity.com/paper/2206.10461/) (2022)
- [TRON: Transformer Neural Network Acceleration with Non-Coherent Silicon Photonics](https://mlsanity.com/paper/2303.12914/) (2023)
- [Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction](https://mlsanity.com/paper/2301.03573/) (2023)
- [Exploring Attention Map Reuse for Efficient Transformer Neural Networks](https://mlsanity.com/paper/2301.12444/) (2023)
- [A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAs](https://mlsanity.com/paper/2411.18148/) (2024)
- [EVE: Environmental Adaptive Neural Network Models for Low-power Energy Harvesting System](https://mlsanity.com/paper/2207.09258/) (2022)
- [PolyMPCNet: Towards ReLU-free Neural Architecture Search in Two-party Computation Based Private Inference](https://mlsanity.com/paper/2209.09424/) (2022)
- [Applying Plain Transformers to Real-World Point Clouds](https://mlsanity.com/paper/2303.00086/) (2023)
- [EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms](https://mlsanity.com/paper/2303.13745/) (2023)
- [TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference](https://mlsanity.com/paper/2303.14882/) (2023)
- [Accelerator-driven Data Arrangement to Minimize Transformers Run-time on Multi-core Architectures](https://mlsanity.com/paper/2312.13000/) (2023)
- [Accelerating Neural Networks for Large Language Models and Graph Processing with Silicon Photonics](https://mlsanity.com/paper/2401.06885/) (2024)
- [ARTEMIS: A Mixed Analog-Stochastic In-DRAM Accelerator for Transformer Neural Networks](https://mlsanity.com/paper/2407.12638/) (2024)
- [ProTEA: Programmable Transformer Encoder Acceleration on FPGA](https://mlsanity.com/paper/2409.13975/) (2024)
- [BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers](https://mlsanity.com/paper/2603.09582/) (2026)
- [BSC: Block-based Stochastic Computing to Enable Accurate and Efficient TinyML](https://mlsanity.com/paper/2111.06686/) (2021)
- [Efficient Traffic State Forecasting using Spatio-Temporal Network Dependencies: A Sparse Graph Neural Network Approach](https://mlsanity.com/paper/2211.03033/) (2022)
- [NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix Operations for Efficient Inference](https://mlsanity.com/paper/2305.14405/) (2023)
- [LoopLynx: A Scalable Dataflow Architecture for Efficient LLM Inference](https://mlsanity.com/paper/2504.09561/) (2025)
- [DxPTA: An Architecture Design Space Exploration with Optical Dataflow-guided Strategy for HW/SW Co-Design of Photonic Transformer Accelerators](https://mlsanity.com/paper/2606.06515/) (2026)

## Abstract (arXiv)

State-of-the-art Transformer-based models, with gigantic parameters, are difficult to be accommodated on resource constrained embedded devices. Moreover, with the development of technology, more and more embedded devices are available to run a Transformer model. For a Transformer model with different constraints (tight or loose), it can be deployed onto devices with different computing power. However, in previous work, designers did not choose the best device among multiple devices. Instead, they just used an existing device to deploy model, which was not necessarily the best fit and may lead to underutilization of resources. To address the deployment challenge of Transformer and the problem to select the best device, we propose an algorithm & hardware closed-loop acceleration framework. Given a dataset, a model, latency constraint LC and accuracy constraint AC, our framework can provide a best device satisfying both constraints. In order to generate a compressed model with high sparsity ratio, we propose a novel pruning technique, hierarchical pruning (HP). We optimize the sparse matrix storage format for HP matrix to further reduce memory usage for FPGA implementation. We design a accelerator that takes advantage of HP to solve the problem of concurrent random access. Experiments on Transformer and TinyBert model show that our framework can find different devices for various LC and AC, covering from low-end devices to high-end devices. Our HP can achieve higher sparsity ratio and is more flexible than other sparsity pattern. Our framework can achieve 37x, 1.9x, 1.7x speedup compared to CPU, GPU and FPGA, respectively.