mlsanityData to 14 Oct 2026

DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation

arXiv 2209.10797 ↗First posted 22 Sep 2022eess.SY cs.AR cs.LGcited by 48

Not on the map. Why?

Cited by 20 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer is a deep learning language model widely used for natural language processing (NLP) services in datacenters. Among transformer models, Generative Pre-trained Transformer (GPT) has achieved remarkable performance in text generation, or natural language generation (NLG), which needs the processing of a large input context in the summarization stage, followed by the generation stage that produces a single word at a time. The conventional platforms such as GPU are specialized for the parallel processing of large inputs in the summarization stage, but their performance significantly degrades in the generation stage due to its sequential characteristic. Therefore, an efficient hardware platform is required to address the high latency caused by the sequential characteristic of text generation. In this paper, we present DFX, a multi-FPGA acceleration appliance that executes GPT-2 model inference end-to-end with low latency and high throughput in both summarization and generation stages. DFX uses model parallelism and optimized dataflow that is model-and-hardware-aware for fast simultaneous workload execution among devices. Its compute cores operate on custom instructions and provide GPT-2 operations end-to-end. We implement the proposed hardware architecture on four Xilinx Alveo U280 FPGAs and utilize all of the channels of the high bandwidth memory (HBM) and the maximum number of compute resources for high hardware efficiency. DFX achieves 5.58x speedup and 3.99x energy efficiency over four NVIDIA V100 GPUs on the modern GPT-2 model. DFX is also 8.21x more cost-effective than the GPU appliance, suggesting that it is a promising solution for text generation workloads in cloud datacenters.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 22 Sep 2022.

Builds on

26 of 53 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2016Layer Normalizationcited by 5,669
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  10. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  11. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  12. 2019Learning Deep Transformer Models for Machine Translationcited by 309
14 more
  1. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  2. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  3. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  4. 2019Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systemscited by 110
  5. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  6. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 101
  7. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 70
  8. 2021A Plug-and-Play Method for Controlled Text Generationcited by 56
  9. 2020Comparing BERT against traditional machine learning text classificationcited by 46
  10. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  11. 2020Efficient Algorithms for Device Placement of DNN Graph Operatorscited by 24
  12. 2016Embracing data abundance: BookTest Dataset for Reading Comprehensioncited by 15
  13. 2020A Sentiment-Controllable Topic-to-Essay Generator with Topic Knowledge Graphcited by 8
  14. 2021First-Generation Inference Accelerator Deployment at Facebookcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 20; 2025: 15; 2026: 8.

48 papers.

Sort:

By area:

  1. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  4. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  5. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  6. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 34
  7. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 33
  8. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 31
  9. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 28
  10. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  11. 2023Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Productioncited by 21
  12. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 20
36 more
  1. 2024A Survey on Hardware Accelerators for Large Language Modelscited by 16
  2. 2024Allo: A Programming Model for Composable Accelerator Designcited by 16
  3. 2024ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameterscited by 14
  4. 2025LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-designcited by 11
  5. 2025PTQTP: Post-Training Quantization to Trit-Planes for Large Language Modelscited by 9
  6. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 9
  7. 2024New Solutions on LLM Acceleration, Optimization, and Applicationcited by 7
  8. 2025Pushing up to the Limit of Memory Bandwidth and Capacity Utilization for Efficient LLM Decoding on Embedded FPGAcited by 7
  9. 2024Ultra Fast Transformers on FPGAs for Particle Physics Experimentscited by 6
  10. 2025TerEffic: Highly Efficient Ternary LLM Inference on FPGAcited by 6
  11. 2024Low Latency Transformer Inference on FPGAs for Physics Applications with hls4mlcited by 5
  12. 2024PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference Systemcited by 5
  13. 2025V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrievalcited by 5
  14. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  15. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  16. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMscited by 4
  17. 2025ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughputcited by 4
  18. 2025MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitivenesscited by 4
  19. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAscited by 4
  20. 2024Hardware Acceleration of LLMs: A comprehensive survey and comparisoncited by 3
  21. 2024TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecturecited by 3
  22. 2025InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNscited by 3
  23. 2026FlexLLM: Composable HLS Library for Flexible Hybrid LLM Accelerator Designcited by 3
  24. 2026A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGAcited by 3
  25. 2023A Dataflow Compiler for Efficient LLM Inference using Custom Microscaling Formatscited by 2
  26. 2026BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Modelscited by 2
  27. 2023YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUscited by 1
  28. 2025AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Designcited by 1
  29. 2025BitStopper: An Efficient Transformer Attention Accelerator via Stage-fusion and Early Terminationcited by 1
  30. 2025LAPA: Log-Domain Prediction-Driven Dynamic Sparsity Accelerator for Transformer Modelcited by 1
  31. 2024The Feasibility of Implementing Large-Scale Transformers on Multi-FPGA Platformscited by 0
  32. 2025LoopLynx: A Scalable Dataflow Architecture for Efficient LLM Inferencecited by 0
  33. 2026SwiftKV: An Edge-Oriented Attention Algorithm and Multi-Head Accelerator for Fast, Efficient LLM Decodingcited by 0
  34. 2026HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusterscited by 0
  35. 2026SymbolicLight V2: Hybrid Neuromorphic Architecture and Sparse Execution for Low-Energy Language Inferencecited by 0
  36. 2026MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Acceleratorscited by 0