# FlexLLM: Composable HLS Library for Flexible Hybrid LLM Accelerator Design

arXiv: https://arxiv.org/abs/2601.15710 · HTML: https://mlsanity.com/paper/2601.15710/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 22 Jan 2026.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing](https://mlsanity.com/paper/2310.08433/) (2023)
- [A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languages](https://mlsanity.com/paper/2410.03981/) (2024)
- [FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs](https://mlsanity.com/paper/2401.03868/) (2024)
- [MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models](https://mlsanity.com/paper/2408.11743/) (2024)
- [DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation](https://mlsanity.com/paper/2209.10797/) (2022)
- [Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference](https://mlsanity.com/paper/2312.15159/) (2023)
- [HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing](https://mlsanity.com/paper/2405.06067/) (2024)
- [EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Models](https://mlsanity.com/paper/2407.21325/) (2024)
- [Allo: A Programming Model for Composable Accelerator Design](https://mlsanity.com/paper/2404.04815/) (2024)
- [SSR: Spatial Sequential Hybrid Architecture for Latency Throughput Tradeoff in Transformer Acceleration](https://mlsanity.com/paper/2401.10417/) (2024)
- [TAPA: A Scalable Task-Parallel Dataflow Programming Framework for Modern FPGAs with Co-Optimization of HLS and Physical Design](https://mlsanity.com/paper/2209.02663/) (2022)
- [InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNs](https://mlsanity.com/paper/2502.08807/) (2025)
- [StreamTensor: Make Tensors Stream in Dataflow Accelerators for LLMs](https://mlsanity.com/paper/2509.13694/) (2025)

## Built on by (most cited first, 3 of 3)

- [A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA](https://mlsanity.com/paper/2603.05931/) (2026)
- [Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference](https://mlsanity.com/paper/2603.29002/) (2026)
- [Can Agents Design Better Chips with a Higher Level Abstraction?](https://mlsanity.com/paper/2609.21157/) (2026)

## Abstract (arXiv)

We present FlexLLM, a composable High-Level Synthesis (HLS) library for rapid development of domain-specific LLM accelerators. FlexLLM exposes key architectural degrees of freedom for stage-customized inference, enabling hybrid designs that tailor temporal reuse and spatial dataflow differently for prefill and decode, and provides a comprehensive quantization suite to support accurate low-bit deployment. Using FlexLLM, we build a complete inference system for the Llama-3.2 1B model in under two months with only 1K lines of code. The system includes: (1) a stage-customized accelerator with hardware-efficient quantization (12.68 WikiText-2 PPL) surpassing SpinQuant baseline, and (2) a Hierarchical Memory Transformer (HMT) plug-in for efficient long-context processing. On the AMD U280 FPGA at 16nm, the accelerator achieves 1.29× end-to-end speedup, 1.64× higher decode throughput, and 3.14× better energy efficiency than an NVIDIA A100 GPU (7nm) running BF16 inference; projected results on the V80 FPGA at 7nm reach 4.71×, 6.55×, and 4.13×, respectively. In long-context scenarios, integrating the HMT plug-in reduces prefill latency by 23.23× and extends the context window by 64×, delivering 1.10×/4.86× lower end-to-end latency and 5.21×/6.27× higher energy efficiency on the U280/V80 compared to the A100 baseline. FlexLLM thus bridges algorithmic innovation in LLM inference and high-performance accelerators with minimal manual effort.