mlsanityData to 7 Oct 2026

Hardware Acceleration of Fully Quantized BERT for Efficient Natural Language Processing

arXiv 2103.02800 ↗First posted 4 Mar 2021cs.AR cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

BERT is the most recent Transformer-based model that achieves state-of-the-art performance in various NLP tasks. In this paper, we investigate the hardware acceleration of BERT on FPGA for edge computing. To tackle the issue of huge computational complexity and memory footprint, we propose to fully quantize the BERT (FQ-BERT), including weights, activations, softmax, layer normalization, and all the intermediate results. Experiments demonstrate that the FQ-BERT can achieve 7.94x compression for weights with negligible performance loss. We then propose an accelerator tailored for the FQ-BERT and evaluate on Xilinx ZCU102 and ZCU111 FPGA. It can achieve a performance-per-watt of 3.18 fps/W, which is 28.91x and 12.72x over Intel(R) Core(TM) i7-8700 CPU and NVIDIA K80 GPU, respectively.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 8 of its 9 references that are in the map. First posted on arXiv 4 Mar 2021.

Builds on

8 of 9 references.

References · 8

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  5. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  6. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  7. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 95
  8. 2019How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representationscited by 63

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3; 2023: 6; 2024: 5; 2025: 5.

21 papers.

Sort:

By area:

  1. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  2. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  3. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 29
  4. 2022VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformercited by 27
  5. 2023SwiftTron: An Efficient Hardware Accelerator for Quantized Transformerscited by 17
  6. 2022Federated Split BERT for Heterogeneous Text Classificationcited by 11
  7. 2023ViTA: A Vision Transformer Inference Accelerator for Edge Applicationscited by 11
  8. 2024HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipelinecited by 7
  9. 2025Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGAcited by 7
  10. 2024BETA: Binarized Energy-Efficient Transformer Accelerator at the Edgecited by 6
  11. 2024P2-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformercited by 4
  12. 2025COBRA: Algorithm-Architecture Co-optimized Binary Transformer Accelerator for Edge Inferencecited by 4
9 more
  1. 2024A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAscited by 3
  2. 2025InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNscited by 3
  3. 2021Vis-TOP: Visual Transformer Overlay Processorcited by 2
  4. 2024Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deploymentcited by 2
  5. 2021Plug-Tagger: A Pluggable Sequence Labeling Framework Using Language Modelscited by 1
  6. 2023Blockwise Compression of Transformer-based Models without Retrainingcited by 1
  7. 2023Block-wise Bit-Compression of Transformer-based Modelscited by 0
  8. 2025LoopLynx: A Scalable Dataflow Architecture for Efficient LLM Inferencecited by 0
  9. 2025Bhasha-Rupantarika: Algorithm-Hardware Co-design approach for Multilingual Neural Machine Translationcited by 0