# EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Models

arXiv: https://arxiv.org/abs/2407.21325 · HTML: https://mlsanity.com/paper/2407.21325/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2024.

## Built on by (most cited first, 21 of 21)

- [Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective](https://mlsanity.com/paper/2410.04466/) (2024)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGA](https://mlsanity.com/paper/2502.10659/) (2025)
- [TerEffic: Highly Efficient Ternary LLM Inference on FPGA](https://mlsanity.com/paper/2502.16473/) (2025)
- [TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge](https://mlsanity.com/paper/2509.13765/) (2025)
- [Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission](https://mlsanity.com/paper/2505.11788/) (2025)
- [SpecMamba: Accelerating Mamba Inference on FPGA with Speculative Decoding](https://mlsanity.com/paper/2509.19873/) (2025)
- [Agent-X: Full Pipeline Acceleration of On-device AI Agents](https://mlsanity.com/paper/2605.10380/) (2026)
- [DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models](https://mlsanity.com/paper/2412.20891/) (2024)
- [FlexLLM: Composable HLS Library for Flexible Hybrid LLM Accelerator Design](https://mlsanity.com/paper/2601.15710/) (2026)
- [Watermarking Large Language Models and the Generated Content: Opportunities and Challenges](https://mlsanity.com/paper/2410.19096/) (2024)
- [TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAs](https://mlsanity.com/paper/2510.15926/) (2025)
- [LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models](https://mlsanity.com/paper/2408.10631/) (2024)
- [AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Design](https://mlsanity.com/paper/2505.03745/) (2025)
- [On the Sustainability of AI Inferences in the Edge](https://mlsanity.com/paper/2507.23093/) (2025)
- [AI Compute Architecture and Evolution Trends](https://mlsanity.com/paper/2508.21394/) (2025)
- [PD-Swap: Prefill-Decode Logic Swapping for End-to-End LLM Inference on Edge FPGAs via Dynamic Partial Reconfiguration](https://mlsanity.com/paper/2512.11550/) (2025)
- [Bhasha-Rupantarika: Algorithm-Hardware Co-design approach for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2510.10676/) (2025)
- [SwiftKV: An Edge-Oriented Attention Algorithm and Multi-Head Accelerator for Fast, Efficient LLM Decoding](https://mlsanity.com/paper/2601.10953/) (2026)
- [HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding](https://mlsanity.com/paper/2607.22389/) (2026)
- [LLMscope: Extracting LLM Assets from Edge AI Chips via Optical Probing](https://mlsanity.com/paper/2608.25321/) (2026)

## Abstract (arXiv)

The rapid advancements in artificial intelligence (AI), particularly the Large Language Models (LLMs), have profoundly affected our daily work and communication forms. However, it is still a challenge to deploy LLMs on resource-constrained edge devices (such as robots), due to the intensive computation requirements, heavy memory access, diverse operator types and difficulties in compilation. In this work, we proposed EdgeLLM to address the above issues. Firstly, focusing on the computation, we designed mix-precision processing element array together with group systolic architecture, that can efficiently support both FP16*FP16 for the MHA block (Multi-Head Attention) and FP16*INT4 for the FFN layer (Feed-Forward Network). Meanwhile specific optimization on log-scale structured weight sparsity, has been used to further increase the efficiency. Secondly, to address the compilation and deployment issue, we analyzed the whole operators within LLM models and developed a universal data parallelism scheme, by which all of the input and output features maintain the same data shape, enabling to process different operators without any data rearrangement. Then we proposed an end-to-end compiler to map the whole LLM model on CPU-FPGA heterogeneous system (AMD Xilinx VCU128 FPGA). The accelerator achieves 1.91x higher throughput and 7.55x higher energy efficiency than the commercial GPU (NVIDIA A100-SXM4-80G). When compared with state-of-the-art FPGA accelerator of FlightLLM, it shows 10-24% better performance in terms of HBM bandwidth utilization, energy efficiency and LLM throughput.