mlsanityData to 14 Oct 2026

Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference

arXiv 2312.15159 ↗First posted 23 Dec 2023cs.LG cs.AI cs.ARcited by 33

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. The majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This paper investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on FPGAs. Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. Through our analysis, we can determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented BERT and GPT2 on an AMD Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4x speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2x speedup compared to DFX, an FPGA overlay, in the prefill stage, while achieving a 1.9x speedup and a 5.7x improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 59 of its 106 references that are in the map. First posted on arXiv 23 Dec 2023.

Builds on

59 of 106 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  13. 2020On Layer Normalization in the Transformer Architecturecited by 922
  14. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  15. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  16. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  17. 2022Efficiently Scaling Transformer Inferencecited by 539
  18. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  19. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  20. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  21. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  22. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  23. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  24. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  25. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  26. 2016FINN: A Framework for Fast, Scalable Binarized Neural Network Inferencecited by 150
  27. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  28. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  29. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  30. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 62
  31. 2018FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networkscited by 56
  32. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  33. 2020FTRANS: Energy-Efficient Acceleration of Transformers using FPGAcited by 54
  34. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 53
  35. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 48
  36. 2020FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activationscited by 32
  37. 2021Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimizationcited by 27
  38. 2021Hardware Acceleration of Fully Quantized BERT for Efficient Natural Language Processingcited by 22

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 16; 2026: 7.

33 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  2. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  3. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  4. 2025Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuningcited by 18
  5. 2024Allo: A Programming Model for Composable Accelerator Designcited by 16
  6. 2025LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-designcited by 11
  7. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  8. 2025TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAscited by 10
  9. 2024HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipelinecited by 7
  10. 2025Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGAcited by 7
  11. 2025TerEffic: Highly Efficient Ternary LLM Inference on FPGAcited by 6
  12. 2025On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Accelerationcited by 6
21 more
  1. 2025Tawa: Automatic Warp Specialization for Modern GPUs with Asynchronous Referencescited by 6
  2. 2025Dato: A Task-Based Programming Model for Dataflow Acceleratorscited by 5
  3. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMscited by 4
  4. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAscited by 4
  5. 2024TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecturecited by 3
  6. 2024A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAscited by 3
  7. 2025InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNscited by 3
  8. 2025End-to-End On-Device Quantization-Aware Training for LLMs at Inference Costcited by 3
  9. 2026FlexLLM: Composable HLS Library for Flexible Hybrid LLM Accelerator Designcited by 3
  10. 2026A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGAcited by 3
  11. 2024LUTMUL: Exceed Conventional FPGA Roofline Limit by LUT-based Efficient Multiplication for Neural Network Inferencecited by 2
  12. 2025NSFlow: An End-to-End FPGA Framework with Scalable Dataflow Architecture for Neuro-Symbolic AIcited by 1
  13. 2025ITERA-LLM: Boosting Sub-8-Bit Large Language Model Inference via Iterative Tensor Decompositioncited by 1
  14. 2025PD-Swap: Prefill-Decode Logic Swapping for End-to-End LLM Inference on Edge FPGAs via Dynamic Partial Reconfigurationcited by 1
  15. 2024The Feasibility of Implementing Large-Scale Transformers on Multi-FPGA Platformscited by 0
  16. 2025LoopLynx: A Scalable Dataflow Architecture for Efficient LLM Inferencecited by 0
  17. 2026Energy Efficient Software Hardware CoDesign for Machine Learning: From TinyML to Large Language Modelscited by 0
  18. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inferencecited by 0
  19. 2026XtraMAC: An Efficient MAC Architecture for Mixed-Precision LLM Inference on FPGAcited by 0
  20. 2026LLMscope: Extracting LLM Assets from Edge AI Chips via Optical Probingcited by 0
  21. 2026SyntheticHLS: Building Diverse Synthetic High-Level Synthesis Datasets using LLMscited by 0