# Hardware Acceleration of LLMs: A comprehensive survey and comparison

arXiv: https://arxiv.org/abs/2409.03384 · HTML: https://mlsanity.com/paper/2409.03384/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 24 of 24)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [A3: Accelerating Attention Mechanisms in Neural Networks with Approximation](https://mlsanity.com/paper/2002.10941/) (2020)
- [TurboTransformers: An Efficient GPU Serving System For Transformer Models](https://mlsanity.com/paper/2010.05680/) (2020)
- [Inference with Reference: Lossless Acceleration of Large Language Models](https://mlsanity.com/paper/2304.04487/) (2023)
- [FTRANS: Energy-Efficient Acceleration of Transformers using FPGA](https://mlsanity.com/paper/2007.08563/) (2020)
- [DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation](https://mlsanity.com/paper/2209.10797/) (2022)
- [ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching](https://mlsanity.com/paper/2403.17312/) (2024)
- [Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer](https://mlsanity.com/paper/2009.08605/) (2020)
- [Simplifying Transformer Blocks](https://mlsanity.com/paper/2311.01906/) (2023)
- [Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attention](https://mlsanity.com/paper/2110.09310/) (2021)
- [X-Former: In-Memory Acceleration of Transformers](https://mlsanity.com/paper/2303.07470/) (2023)
- [AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformers](https://mlsanity.com/paper/2302.14705/) (2023)
- [LightSeq2: Accelerated Training for Transformer-based Models on GPUs](https://mlsanity.com/paper/2110.05722/) (2021)
- [NPE: An FPGA-based Overlay Processor for Natural Language Processing](https://mlsanity.com/paper/2104.06535/) (2021)
- [SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences](https://mlsanity.com/paper/2206.14550/) (2022)
- [SSR: Spatial Sequential Hybrid Architecture for Latency Throughput Tradeoff in Transformer Acceleration](https://mlsanity.com/paper/2401.10417/) (2024)
- [Transformer-Based Autonomous Driving Models and Deployment-Oriented Compression: A Survey](https://mlsanity.com/paper/2304.10891/) (2023)
- [A Comprehensive Performance Study of Large Language Models on Novel AI Accelerators](https://mlsanity.com/paper/2310.04607/) (2023)
- [TransAxx: Efficient Transformers with Approximate Computing](https://mlsanity.com/paper/2402.07545/) (2024)
- [ME-ViT: A Single-Load Memory-Efficient FPGA Accelerator for Vision Transformers](https://mlsanity.com/paper/2402.09709/) (2024)
- [BETA: Binarized Energy-Efficient Transformer Accelerator at the Edge](https://mlsanity.com/paper/2401.11851/) (2024)
- [An Efficient FPGA-Based Accelerator for Swin Transformer](https://mlsanity.com/paper/2308.13922/) (2023)
- [A Cost-Efficient FPGA Implementation of Tiny Transformer Model using Neural ODE](https://mlsanity.com/paper/2401.02721/) (2024)

## Built on by (most cited first, 3 of 3)

- [Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective](https://mlsanity.com/paper/2410.04466/) (2024)
- [Gated Differentiable Working Memory for Long-Context Language Modeling](https://mlsanity.com/paper/2601.12906/) (2026)
- [Recent Developments in Transformer Inference Deployment on FPGA Platforms: A Survey](https://mlsanity.com/paper/2609.01212/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have emerged as powerful tools for natural language processing tasks, revolutionizing the field with their ability to understand and generate human-like text. In this paper, we present a comprehensive survey of the several research efforts that have been presented for the acceleration of transformer networks for Large Language Models using hardware accelerators. The survey presents the frameworks that have been proposed and then performs a qualitative and quantitative comparison regarding the technology, the processing platform (FPGA, ASIC, In-Memory, GPU), the speedup, the energy efficiency, the performance (GOPs), and the energy efficiency (GOPs/W) of each framework. The main challenge in comparison is that every proposed scheme is implemented on a different process technology making hard a fair comparison. The main contribution of this paper is that we extrapolate the results of the performance and the energy efficiency on the same technology to make a fair comparison; one theoretical and one more practical. We implement part of the LLMs on several FPGA chips to extrapolate the results to the same process technology and then we make a fair comparison of the performance.