mlsanityData to 14 Oct 2026

HPU: High-Bandwidth Processing Unit for Scalable, Cost-effective LLM Inference via GPU Co-processing

arXiv 2504.16112 ↗First posted 18 Apr 2025cs.AR cs.AI cs.CLcited by 5

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The attention layer, a core component of Transformer-based LLMs, brings out inefficiencies in current GPU systems due to its low operational intensity and the substantial memory requirements of KV caches. We propose a High-bandwidth Processing Unit (HPU), a memoryintensive co-processor that enhances GPU resource utilization during large-batched LLM inference. By offloading memory-bound operations, the HPU allows the GPU to focus on compute-intensive tasks, increasing overall efficiency. Also, the HPU, as an add-on card, scales out to accommodate surging memory demands driven by large batch sizes and extended sequence lengths. In this paper, we show the HPU prototype implemented with PCIe-based FPGA cards mounted on a GPU system. Our novel GPU-HPU heterogeneous system demonstrates up to 4.1x performance gains and 4.6x energy efficiency improvements over a GPUonly system, providing scalability without increasing the number of GPUs.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 8 of its 17 references that are in the map. First posted on arXiv 18 Apr 2025.

Builds on

8 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines18 Mar

Other references · 7

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  6. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 52
  7. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIMcited by 5
  2. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMscited by 5
  3. 2026Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compressioncited by 2
  4. 2025DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architecturescited by 0
  5. 2026NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient LLM inference solution on Intel GPU
  2. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
  3. 2026FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference
  4. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  5. 2025Make LLM Inference Affordable to Everyone: Augmenting GPU Memory with NDP-DIMM
  6. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  7. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  8. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving