# A Survey on Hardware Accelerators for Large Language Models

arXiv: https://arxiv.org/abs/2401.09890 · HTML: https://mlsanity.com/paper/2401.09890/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Builds on (most cited first, 14 of 14)

- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://mlsanity.com/paper/2312.11514/) (2023)
- [Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://mlsanity.com/paper/2203.16487/) (2022)
- [A3: Accelerating Attention Mechanisms in Neural Networks with Approximation](https://mlsanity.com/paper/2002.10941/) (2020)
- [FTRANS: Energy-Efficient Acceleration of Transformers using FPGA](https://mlsanity.com/paper/2007.08563/) (2020)
- [DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation](https://mlsanity.com/paper/2209.10797/) (2022)
- [Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer](https://mlsanity.com/paper/2009.08605/) (2020)
- [Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attention](https://mlsanity.com/paper/2110.09310/) (2021)
- [X-Former: In-Memory Acceleration of Transformers](https://mlsanity.com/paper/2303.07470/) (2023)
- [LightSeq2: Accelerated Training for Transformer-based Models on GPUs](https://mlsanity.com/paper/2110.05722/) (2021)
- [NPE: An FPGA-based Overlay Processor for Natural Language Processing](https://mlsanity.com/paper/2104.06535/) (2021)
- [Transformer-Based Autonomous Driving Models and Deployment-Oriented Compression: A Survey](https://mlsanity.com/paper/2304.10891/) (2023)
- [A Comprehensive Performance Study of Large Language Models on Novel AI Accelerators](https://mlsanity.com/paper/2310.04607/) (2023)

## Built on by (most cited first, 16 of 16)

- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective](https://mlsanity.com/paper/2410.04466/) (2024)
- [Designing Efficient LLM Accelerators for Edge Devices](https://mlsanity.com/paper/2408.00462/) (2024)
- [A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency](https://mlsanity.com/paper/2505.01658/) (2025)
- [CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs](https://mlsanity.com/paper/2409.12490/) (2024)
- [Deploying Foundation Model Powered Agent Services: A Survey](https://mlsanity.com/paper/2412.13437/) (2024)
- [PIM-AI: A Novel Architecture for High-Efficiency LLM Inference](https://mlsanity.com/paper/2411.17309/) (2024)
- [Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI](https://mlsanity.com/paper/2603.07466/) (2026)
- [Multimodal Data Storage and Retrieval for Embodied AI: A Survey](https://mlsanity.com/paper/2508.13901/) (2025)
- [Energy Efficient Software Hardware CoDesign for Machine Learning: From TinyML to Large Language Models](https://mlsanity.com/paper/2603.23668/) (2026)
- [Focus Session: Hardware and Software Techniques for Accelerating Multimodal Foundation Models](https://mlsanity.com/paper/2604.21952/) (2026)
- [You Only Charge Once 2.0 : A End-to-End Analog Computing-in-Memory Architecture with Reconfigurable Switched Capacitors](https://mlsanity.com/paper/2608.11116/) (2026)
- [Recent Developments in Transformer Inference Deployment on FPGA Platforms: A Survey](https://mlsanity.com/paper/2609.01212/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have emerged as powerful tools for natural language processing tasks, revolutionizing the field with their ability to understand and generate human-like text. As the demand for more sophisticated LLMs continues to grow, there is a pressing need to address the computational challenges associated with their scale and complexity. This paper presents a comprehensive survey on hardware accelerators designed to enhance the performance and energy efficiency of Large Language Models. By examining a diverse range of accelerators, including GPUs, FPGAs, and custom-designed architectures, we explore the landscape of hardware solutions tailored to meet the unique computational demands of LLMs. The survey encompasses an in-depth analysis of architecture, performance metrics, and energy efficiency considerations, providing valuable insights for researchers, engineers, and decision-makers aiming to optimize the deployment of LLMs in real-world applications.