mlsanityData to 7 Oct 2026

PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Serving

arXiv 2501.08192 ↗First posted 14 Jan 2025cs.AI cs.AR cs.DCcited by 9

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) are typically served from clusters of GPUs/NPUs that consist of large number of devices. Unfortunately, communication between these devices incurs significant overhead, increasing the inference latency and cost while limiting the scalability. Prior work addressed this issue by overlapping communication with compute, but has severe limitations due to the data dependencies between these operations. In this paper, we propose PRESERVE, a novel framework that prefetches model weights and KV-cache from off-chip HBM memory to the on-chip cache of AI accelerators during the communication operations, which offers various advantages and performance improvements compared to prior methods. Through extensive experiments conducted on commercial AI accelerators, we demonstrate up to 1.6x end-to-end speedup on state-of-the-art, open-source LLMs. Additionally, we perform a design space exploration that identifies the optimal hardware configuration for the proposed method, showing a further 1.25x improvement in performance per cost by selecting the optimal L2 cache size. Our results show that PRESERVE has the potential to mitigate the memory bottlenecks and communication overheads, offering a solution to improve the performance and scalability of the LLM inference systems.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 41 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

17 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2024Mixtral of Expertscited by 1,785
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  11. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  12. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 53
4 more
  1. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  2. 2020Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platformscited by 25
  3. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20
  4. 2023Optimizing Distributed ML Communication with Fused Computation-Collective Operationscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  2. 2025Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetchingcited by 4
  3. 2025Architecting Long-Context LLM Acceleration with Packing-Prefetch Scheduler and Ultra-Large Capacity On-Chip Memoriescited by 2
  4. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  5. 2025AI Compute Architecture and Evolution Trendscited by 1
  6. 2025EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUscited by 1
  7. 2026HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architecturescited by 0
  8. 2026NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetchingcited by 0
  9. 2026FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  3. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  6. 2024Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching
  7. 2024KVDirect: Distributed Disaggregated LLM Inference
  8. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation