FAMOUS: Flexible Accelerator for the Attention Mechanism of Transformer on UltraScale+ FPGAs
Not on the map. Why?
Abstract (arXiv)
Transformer neural networks (TNNs) are being applied across a widening range of application domains, including natural language processing (NLP), machine translation, and computer vision (CV). Their popularity is largely attributed to the exceptional performance of their multi-head self-attention blocks when analyzing sequential data and extracting features. To date, there are limited hardware accelerators tailored for this mechanism, which is the first step before designing an accelerator for a complete model. This paper proposes textitFAMOUS, a flexible hardware accelerator for dense multi-head attention (MHA) computation of TNNs on field-programmable gate arrays (FPGAs). It is optimized for high utilization of processing elements and on-chip memories to improve parallelism and reduce latency. An efficient tiling of large matrices has been employed to distribute memory and computing resources across different modules on various FPGA platforms. The design is evaluated on Xilinx Alveo U55C and U200 data center cards containing Ultrascale+ FPGAs. Experimental results are presented that show that it can attain a maximum throughput, number of parallel attention heads, embedding dimension and tile size of 328 (giga operations/second (GOPS)), 8, 768 and 64 respectively on the U55C. Furthermore, it is 3.28× and 2.6× faster than the Intel Xeon Gold 5220R CPU and NVIDIA V100 GPU respectively. It is also 1.3× faster than the fastest state-of-the-art FPGA-based accelerator.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 45 references that are in the map. First posted on arXiv 21 Sep 2024.
Builds on
18 of 45 references.
References · 18
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
- 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
- 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
- 2019StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understandingcited by 108
- 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 97
- 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
- 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 60
6 more
- 2020FTRANS: Energy-Efficient Acceleration of Transformers using FPGAcited by 52
- 2020Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformercited by 42
- 2022A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipeliningcited by 21
- 2022SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequencescited by 10
- 2024Ultra Fast Transformers on FPGAs for Particle Physics Experimentscited by 6
- 2023Accelerating LSTM-based High-Rate Dynamic System Modelscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.