mlsanityData to 7 Oct 2026

KVDirect: Distributed Disaggregated LLM Inference

arXiv 2501.14743 ↗First posted 13 Dec 2024cs.DC cs.LG cs.PFcited by 8

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Models (LLMs) have become the new foundation for many applications, reshaping human society like a storm. Disaggregated inference, which separates prefill and decode stages, is a promising approach to improving hardware utilization and service quality. However, due to inefficient inter-node communication, existing systems restrict disaggregated inference to a single node, limiting resource allocation flexibility and reducing service capacity. This paper introduces KVDirect, which optimizes KV cache transfer to enable a distributed disaggregated LLM inference. KVDirect achieves this through the following contributions. First, we propose a novel tensor-centric communication mechanism that reduces the synchronization overhead in traditional distributed GPU systems. Second, we design a custom communication library to support dynamic GPU resource scheduling and efficient KV cache transfer. Third, we introduce a pull-based KV cache transfer strategy that reduces GPU resource idling and improves latency. Finally, we implement KVDirect as an open-source LLM inference framework. Our evaluation demonstrates that KVDirect reduces per-request latency by 55% compared to the baseline across diverse workloads under the same resource constraints.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 49 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

31 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  3. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  12. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
15 more
  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  2. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  3. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  4. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  5. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  6. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  7. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  8. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  9. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  10. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  11. 2023Generative Sequential Recommendation with GPTReccited by 21
  12. 2023Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameterscited by 13
  13. 2022No Provisioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computingcited by 8
  14. 2024Crafting Knowledge: Exploring the Creative Mechanisms of Chat-Based Search Enginescited by 5
  15. 2022Replicating Persistent Memory Key-Value Stores with Efficient RDMA Abstractioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Managementcited by 11
  2. 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Schedulingcited by 10
  3. 2025Beyond the Buzz: A Pragmatic Take on Inference Disaggregationcited by 9
  4. 2025Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inferencecited by 5
  5. 2025Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUscited by 3
  6. 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructurecited by 3
  7. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXLcited by 2
  8. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
  2. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  3. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  4. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  5. 2026A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
  6. 2023Efficient LLM inference solution on Intel GPU
  7. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
  8. 2026Lodestar: An Online-Learning LLM Inference Router