# Hardware Acceleration of Fully Quantized BERT for Efficient Natural Language Processing

arXiv: https://arxiv.org/abs/2103.02800 · HTML: https://mlsanity.com/paper/2103.02800/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 9 references that are in the map.
- First posted on arXiv 4 Mar 2021.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks](https://mlsanity.com/paper/1712.01507/) (2017)
- [How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations](https://mlsanity.com/paper/1909.04925/) (2019)

## Built on by (most cited first, 22 of 22)

- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference](https://mlsanity.com/paper/2312.15159/) (2023)
- [VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer](https://mlsanity.com/paper/2201.06618/) (2022)
- [SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers](https://mlsanity.com/paper/2304.03986/) (2023)
- [Federated Split BERT for Heterogeneous Text Classification](https://mlsanity.com/paper/2205.13299/) (2022)
- [ViTA: A Vision Transformer Inference Accelerator for Edge Applications](https://mlsanity.com/paper/2302.09108/) (2023)
- [HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline](https://mlsanity.com/paper/2407.17879/) (2024)
- [Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGA](https://mlsanity.com/paper/2502.10659/) (2025)
- [BETA: Binarized Energy-Efficient Transformer Accelerator at the Edge](https://mlsanity.com/paper/2401.11851/) (2024)
- [P2-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformer](https://mlsanity.com/paper/2405.19915/) (2024)
- [COBRA: Algorithm-Architecture Co-optimized Binary Transformer Accelerator for Edge Inference](https://mlsanity.com/paper/2504.16269/) (2025)
- [TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecture](https://mlsanity.com/paper/2411.03697/) (2024)
- [A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAs](https://mlsanity.com/paper/2411.18148/) (2024)
- [InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNs](https://mlsanity.com/paper/2502.08807/) (2025)
- [Vis-TOP: Visual Transformer Overlay Processor](https://mlsanity.com/paper/2110.10957/) (2021)
- [Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deployment](https://mlsanity.com/paper/2407.12070/) (2024)
- [Plug-Tagger: A Pluggable Sequence Labeling Framework Using Language Models](https://mlsanity.com/paper/2110.07331/) (2021)
- [Blockwise Compression of Transformer-based Models without Retraining](https://mlsanity.com/paper/2304.01483/) (2023)
- [Block-wise Bit-Compression of Transformer-based Models](https://mlsanity.com/paper/2303.09184/) (2023)
- [LoopLynx: A Scalable Dataflow Architecture for Efficient LLM Inference](https://mlsanity.com/paper/2504.09561/) (2025)
- [Bhasha-Rupantarika: Algorithm-Hardware Co-design approach for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2510.10676/) (2025)

## Abstract (arXiv)

BERT is the most recent Transformer-based model that achieves state-of-the-art performance in various NLP tasks. In this paper, we investigate the hardware acceleration of BERT on FPGA for edge computing. To tackle the issue of huge computational complexity and memory footprint, we propose to fully quantize the BERT (FQ-BERT), including weights, activations, softmax, layer normalization, and all the intermediate results. Experiments demonstrate that the FQ-BERT can achieve 7.94x compression for weights with negligible performance loss. We then propose an accelerator tailored for the FQ-BERT and evaluate on Xilinx ZCU102 and ZCU111 FPGA. It can achieve a performance-per-watt of 3.18 fps/W, which is 28.91x and 12.72x over Intel(R) Core(TM) i7-8700 CPU and NVIDIA K80 GPU, respectively.