# FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs

arXiv: https://arxiv.org/abs/2401.03868 · HTML: https://mlsanity.com/paper/2401.03868/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 28 of its 52 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [A3: Accelerating Attention Mechanisms in Neural Networks with Approximation](https://mlsanity.com/paper/2002.10941/) (2020)
- [FTRANS: Energy-Efficient Acceleration of Transformers using FPGA](https://mlsanity.com/paper/2007.08563/) (2020)
- [FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks](https://mlsanity.com/paper/2107.06419/) (2021)
- [FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activations](https://mlsanity.com/paper/2012.12206/) (2020)
- [Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design](https://mlsanity.com/paper/2209.09570/) (2022)
- [A Study on the Implementation of Generative AI Services Using an Enterprise Data-Based LLM Application Architecture](https://mlsanity.com/paper/2309.01105/) (2023)
- [Dynamic N:M Fine-grained Structured Sparse Attention Mechanism](https://mlsanity.com/paper/2203.00091/) (2022)
- [Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences](https://mlsanity.com/paper/2210.11794/) (2022)
- [Benchmarking High Bandwidth Memory on FPGAs](https://mlsanity.com/paper/2005.04324/) (2020)
- [Logic Shrinkage: Learned FPGA Netlist Sparsity for Efficient Neural Network Inference](https://mlsanity.com/paper/2112.02346/) (2021)
- [N3H-Core: Neuron-designed Neural Network Accelerator via FPGA-based Heterogeneous Computing Cores](https://mlsanity.com/paper/2112.08193/) (2021)

## Built on by (most cited first, 30 of 64)

- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines](https://mlsanity.com/paper/2403.11421/) (2024)
- [Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective](https://mlsanity.com/paper/2410.04466/) (2024)
- [A Survey of Small Language Models](https://mlsanity.com/paper/2410.20011/) (2024)
- [Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning](https://mlsanity.com/paper/2406.07081/) (2024)
- [LLM-Powered AI Agent Systems and Their Applications in Industry](https://mlsanity.com/paper/2505.16120/) (2025)
- [Progressive Mixed-Precision Decoding for Efficient LLM Inference](https://mlsanity.com/paper/2410.13461/) (2024)
- [Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning](https://mlsanity.com/paper/2502.03304/) (2025)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2406.08413/) (2024)
- [Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization](https://mlsanity.com/paper/2503.18599/) (2025)
- [LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-design](https://mlsanity.com/paper/2502.15260/) (2025)
- [Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Training](https://mlsanity.com/paper/2504.20314/) (2025)
- [PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models](https://mlsanity.com/paper/2509.16989/) (2025)
- [Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies](https://mlsanity.com/paper/2603.16952/) (2026)
- [New Solutions on LLM Acceleration, Optimization, and Application](https://mlsanity.com/paper/2406.10903/) (2024)
- [Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGA](https://mlsanity.com/paper/2502.10659/) (2025)
- [TerEffic: Highly Efficient Ternary LLM Inference on FPGA](https://mlsanity.com/paper/2502.16473/) (2025)
- [TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge](https://mlsanity.com/paper/2509.13765/) (2025)
- [Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models](https://mlsanity.com/paper/2509.18763/) (2025)
- [SpecMamba: Accelerating Mamba Inference on FPGA with Speculative Decoding](https://mlsanity.com/paper/2509.19873/) (2025)
- [KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware](https://mlsanity.com/paper/2603.08721/) (2026)
- [Deploying Foundation Model Powered Agent Services: A Survey](https://mlsanity.com/paper/2412.13437/) (2024)
- [MEADOW: Memory-efficient Dataflow and Data Packing for Low Power Edge LLMs](https://mlsanity.com/paper/2503.11663/) (2025)
- [LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs](https://mlsanity.com/paper/2511.06174/) (2025)
- [TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecture](https://mlsanity.com/paper/2411.03697/) (2024)
- [A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAs](https://mlsanity.com/paper/2411.18148/) (2024)
- [InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNs](https://mlsanity.com/paper/2502.08807/) (2025)
- [End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost](https://mlsanity.com/paper/2509.00031/) (2025)

## Abstract (arXiv)

Transformer-based Large Language Models (LLMs) have made a significant impact on various domains. However, LLMs' efficiency suffers from both heavy computation and memory overheads. Compression techniques like sparsification and quantization are commonly used to mitigate the gap between LLM's computation/memory overheads and hardware capacity. However, existing GPU and transformer-based accelerators cannot efficiently process compressed LLMs, due to the following unresolved challenges: low computational efficiency, underutilized memory bandwidth, and large compilation overheads. This paper proposes FlightLLM, enabling efficient LLMs inference with a complete mapping flow on FPGAs. In FlightLLM, we highlight an innovative solution that the computation and memory overhead of LLMs can be solved by utilizing FPGA-specific resources (e.g., DSP48 and heterogeneous memory hierarchy). We propose a configurable sparse DSP chain to support different sparsity patterns with high computation efficiency. Second, we propose an always-on-chip decode scheme to boost memory bandwidth with mixed-precision support. Finally, to make FlightLLM available for real-world LLMs, we propose a length adaptive compilation method to reduce the compilation overhead. Implemented on the Xilinx Alveo U280 FPGA, FlightLLM achieves 6.0× higher energy efficiency and 1.8× better cost efficiency against commercial GPUs (e.g., NVIDIA V100S) on modern LLMs (e.g., LLaMA2-7B) using vLLM and SmoothQuant under the batch size of one. FlightLLM beats NVIDIA A100 GPU with 1.2× higher throughput using the latest Versal VHK158 FPGA.