Data to 7 Oct 2026

D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving

arXiv 2403.01876 ↗First posted 4 Mar 2024cs.DCcited by 48

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Distributed LLM serving is costly and often underutilizes hardware accelerators due to three key challenges: bubbles in pipeline-parallel deployments caused by the bimodal latency of prompt and token processing, GPU memory overprovisioning, and long recovery times in case of failures. In this paper, we propose D'ej`aVu, a system to address all these challenges using a versatile and efficient KV cache streaming library (D'ej`aVuLib). Using D'ej`aVuLib, we propose and implement efficient prompt-token disaggregation to reduce pipeline bubbles, microbatch swapping for efficient GPU memory management, and state replication for fault-tolerance. We highlight the efficacy of these solutions on a range of large models across cloud deployments.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 37 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

17 of 37 references.

Closest ancestors · 1

Same direction, newest first

2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 16

2020Language Models are Few-Shot Learnerscited by 32,199
2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
2022Efficiently Scaling Transformer Inferencecited by 529
2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79

4 more

2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 55
2021FastSeq: Make Sequence Generation Fastercited by 13
2022Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Serverscited by 6
2023Packrat: Automatic Reconfiguration for Latency Minimization in CPU-based DNN Servingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 18; 2026: 16.

48 papers.

Sort:

By area:

2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
2024Efficient LLM Scheduling by Learning to Rankcited by 83
2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Servingcited by 11

36 more

2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 11
2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inferencecited by 10
2025Beyond the Buzz: A Pragmatic Take on Inference Disaggregationcited by 9
2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8
2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashingcited by 6
2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
2025Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inferencecited by 5
2025Reliable and Resilient Collective Communication Library for LLM Training and Servingcited by 4
2026Towards Resiliency in Large Language Model Serving with KevlarFlowcited by 3
2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Servingcited by 3
2024On the Cost of Model-Serving Frameworks: An Experimental Evaluationcited by 2
2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systemscited by 2
2025MatKV: Trading Compute for Flash Storage in LLM Inferencecited by 2
2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
2024BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batchingcited by 1
2025Good things come in small packages: Should we build AI clusters with Lite-GPUs?cited by 1
2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 1
2025MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignmentcited by 1
2025Towards Sampling Data Structures for Tensor Products in Turnstile Streamscited by 1
2025Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharingcited by 1
2026LUMEN: Coordinated Failure Recovery for Distributed LLM Servingcited by 1
2026DiLaServe: High SLO Attainment Serving for Diffusion Language Modelscited by 1
2026Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Servingcited by 1
2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMVcited by 0
2025CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperationcited by 0
2025OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latencycited by 0
2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 0
2026Scalable AI Inference: Performance Analysis and Optimization of AI Model Servingcited by 0
2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemptioncited by 0
2026Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexingcited by 0
2026TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructurecited by 0
2026Failure-Aware Long-Form Translation: Design and Implementation of a Recoverable LLM Translation Systemcited by 0
2026Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamocited by 0

Similar, not cited

Close in wording, no citation either way