# ME-ViT: A Single-Load Memory-Efficient FPGA Accelerator for Vision Transformers

arXiv: https://arxiv.org/abs/2402.09709 · HTML: https://mlsanity.com/paper/2402.09709/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2024.

## Built on by (most cited first, 7 of 7)

- [Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies](https://mlsanity.com/paper/2503.02891/) (2025)
- [Hardware Acceleration of LLMs: A comprehensive survey and comparison](https://mlsanity.com/paper/2409.03384/) (2024)
- [TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecture](https://mlsanity.com/paper/2411.03697/) (2024)
- [LL-ViT: Edge Deployable Vision Transformers with Look Up Table Neurons](https://mlsanity.com/paper/2511.00812/) (2025)
- [CHOSEN: Compilation to Hardware Optimization Stack for Efficient Vision Transformer Inference](https://mlsanity.com/paper/2407.12736/) (2024)
- [Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation](https://mlsanity.com/paper/2605.14047/) (2026)
- [FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers](https://mlsanity.com/paper/2606.31938/) (2026)

## Abstract (arXiv)

Vision Transformers (ViTs) have emerged as a state-of-the-art solution for object classification tasks. However, their computational demands and high parameter count make them unsuitable for real-time inference, prompting the need for efficient hardware implementations. Existing hardware accelerators for ViTs suffer from frequent off-chip memory access, restricting the achievable throughput by memory bandwidth. In devices with a high compute-to-communication ratio (e.g., edge FPGAs with limited bandwidth), off-chip memory access imposes a severe bottleneck on overall throughput. This work proposes ME-ViT, a novel underlineMemory underlineEfficient FPGA accelerator for underlineViT inference that minimizes memory traffic. We propose a textitsingle-load policy in designing ME-ViT: model parameters are only loaded once, intermediate results are stored on-chip, and all operations are implemented in a single processing element. To achieve this goal, we design a memory-efficient processing element (ME-PE), which processes multiple key operations of ViT inference on the same architecture through the reuse of textitmulti-purpose buffers. We also integrate the Softmax and LayerNorm functions into the ME-PE, minimizing stalls between matrix multiplications. We evaluate ME-ViT on systolic array sizes of 32 and 16, achieving up to a 9.22× and 17.89× overall improvement in memory bandwidth, and a 2.16× improvement in throughput per DSP for both designs over state-of-the-art ViT accelerators on FPGA. ME-ViT achieves a power efficiency improvement of up to 4.00× (1.03×) over a GPU (FPGA) baseline. ME-ViT enables up to 5 ME-PE instantiations on a Xilinx Alveo U200, achieving a 5.10× improvement in throughput over the state-of-the art FPGA baseline, and a 5.85× (1.51×) improvement in power efficiency over the GPU (FPGA) baseline.