mlsanityData to 13 Oct 2026

FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines

arXiv 2403.11421 ↗First posted 18 Mar 2024cs.DC cs.PFcited by 49

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Cost of serving large language models (LLM) is high, but the expensive and scarce GPUs are poorly efficient when generating tokens sequentially, unless the batch of sequences is enlarged. However, the batch size is limited by some constantly reused intermediate results, namely KV-Cache. They occupy too much memory to fit more sequences into a GPU simultaneously. While they could be offloaded to host memory, the CPU-GPU bandwidth is an inevitable bottleneck. We find a way to decompose the transformer models into two parts of different characteristics, one of which includes the memory-bound KV-Cache accessing. Our key insight is that the aggregated memory capacity, bandwidth, and computing power of CPUs across multiple nodes is an efficient option to process this part. Performance improvement comes from reduced data transmission overhead and boosted GPU throughput to process the other model part. Moreover, we address efficiency challenges brought by heterogeneity at both temporal and inter-device scopes using scheduling and performance modeling techniques. Evaluation results show that our system achieves 1.88x - 5.04x the throughput of vLLM when serving modern LLMs with the same GPU.

Key facts. Cited by 49 papers in the map (data through 13 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 34 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

25 of 34 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs8 Jan
  2. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan
  3. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov

Other references · 22

  1. 2017Attention Is All You Needcited by 60,106
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  4. 2023GPT-4 Technical Reportcited by 18,278
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  10. 2020Longformer: The Long-Document Transformercited by 3,393
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
10 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,884
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,272
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 626
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 570
  5. 2022Efficiently Scaling Transformer Inferencecited by 532
  6. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 213
  7. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  8. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  9. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  10. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 33

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 12; 2025: 26; 2026: 11.

49 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 202
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 131
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 126
  4. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  6. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  7. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  8. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 32
  9. 2025Cognitive Memory in Large Language Modelscited by 29
  10. 2025Tensor Product Attention Is All You Needcited by 28
  11. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  12. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
37 more
  1. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 18
  2. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  3. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  4. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregationcited by 14
  5. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputationcited by 12
  6. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  7. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  8. 2025Hardware-Efficient Attention for Fast Decodingcited by 10
  9. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 10
  10. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9
  11. 2025Beyond the Buzz: A Pragmatic Take on Inference Disaggregationcited by 9
  12. 2024PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference Systemcited by 5
  13. 2025HPU: High-Bandwidth Processing Unit for Scalable, Cost-effective LLM Inference via GPU Co-processingcited by 5
  14. 2025CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIMcited by 5
  15. 2025KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embeddingcited by 5
  16. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  17. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  18. 2025SSR: Speculative Parallel Scaling Reasoning in Test-timecited by 4
  19. 2025Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Servingcited by 3
  20. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMscited by 3
  21. 2025FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoningcited by 3
  22. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 3
  23. 2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchipscited by 2
  24. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2
  25. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Managementcited by 2
  26. 2026GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inferencecited by 2
  27. 2025FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concerncited by 1
  28. 2025Argus: Quality-Aware High-Throughput Text-to-Image Inference Serving Systemcited by 1
  29. 2025Round Attention: A Novel Round-Level Attention Mechanism to Accelerate LLM Inferencecited by 0
  30. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUscited by 0
  31. 2025Learn from the Past: Fast Sparse Indexing for Large Language Model Decodingcited by 0
  32. 2025CTkvr: KV Cache Retrieval for Long-Context LLMs via Centroid then Token Indexingcited by 0
  33. 2026Understanding Rate-Distortion Performance in Distributed Transformer Inferencecited by 0
  34. 2026Multi-Head Low-Rank Attentioncited by 0
  35. 2026Communication-Efficient Verifiable Attention for LLM Inferencecited by 0
  36. 2026Training-Free Hashing-Based Attention via Binary Principal Componentscited by 0
  37. 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  2. 2023Efficient LLM inference solution on Intel GPU
  3. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
  4. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow
  5. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  6. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
  7. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management
  8. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving