# NPE: An FPGA-based Overlay Processor for Natural Language Processing

arXiv: https://arxiv.org/abs/2104.06535 · HTML: https://mlsanity.com/paper/2104.06535/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2021.

## Built on by (most cited first, 17 of 17)

- [Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference](https://mlsanity.com/paper/2403.20306/) (2024)
- [Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference](https://mlsanity.com/paper/2312.15159/) (2023)
- [A Survey on Hardware Accelerators for Large Language Models](https://mlsanity.com/paper/2401.09890/) (2024)
- [Designing Efficient LLM Accelerators for Edge Devices](https://mlsanity.com/paper/2408.00462/) (2024)
- [TerEffic: Highly Efficient Ternary LLM Inference on FPGA](https://mlsanity.com/paper/2502.16473/) (2025)
- [Deploying Foundation Model Powered Agent Services: A Survey](https://mlsanity.com/paper/2412.13437/) (2024)
- [A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs](https://mlsanity.com/paper/2502.09921/) (2025)
- [Hardware Acceleration of LLMs: A comprehensive survey and comparison](https://mlsanity.com/paper/2409.03384/) (2024)
- [TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecture](https://mlsanity.com/paper/2411.03697/) (2024)
- [Green Prompting: Characterizing Prompt-driven Energy Costs of LLM Inference](https://mlsanity.com/paper/2503.10666/) (2025)
- [Vis-TOP: Visual Transformer Overlay Processor](https://mlsanity.com/paper/2110.10957/) (2021)
- [HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis](https://mlsanity.com/paper/2405.00738/) (2024)
- [NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix Operations for Efficient Inference](https://mlsanity.com/paper/2305.14405/) (2023)
- [The Feasibility of Implementing Large-Scale Transformers on Multi-FPGA Platforms](https://mlsanity.com/paper/2404.16158/) (2024)
- [LoopLynx: A Scalable Dataflow Architecture for Efficient LLM Inference](https://mlsanity.com/paper/2504.09561/) (2025)
- [F-BFQ: Flexible Block Floating-Point Quantization Accelerator for LLMs](https://mlsanity.com/paper/2510.13401/) (2025)
- [ELiTeFormer: An Efficient Transformer for FPGAs](https://mlsanity.com/paper/2607.03652/) (2026)

## Abstract (arXiv)

In recent years, transformer-based models have shown state-of-the-art results for Natural Language Processing (NLP). In particular, the introduction of the BERT language model brought with it breakthroughs in tasks such as question answering and natural language inference, advancing applications that allow humans to interact naturally with embedded devices. FPGA-based overlay processors have been shown as effective solutions for edge image and video processing applications, which mostly rely on low precision linear matrix operations. In contrast, transformer-based NLP techniques employ a variety of higher precision nonlinear operations with significantly higher frequency. We present NPE, an FPGA-based overlay processor that can efficiently execute a variety of NLP models. NPE offers software-like programmability to the end user and, unlike FPGA designs that implement specialized accelerators for each nonlinear function, can be upgraded for future NLP models without requiring reconfiguration. We demonstrate that NPE can meet real-time conversational AI latency targets for the BERT language model with 4× lower power than CPUs and 6× lower power than GPUs. We also show NPE uses 3× fewer FPGA resources relative to comparable BERT network-specific accelerators in the literature. NPE provides a cost-effective and power-efficient FPGA-based solution for Natural Language Processing at the edge.