mlsanityData to 7 Oct 2026

Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference

arXiv 2602.00328 ↗First posted 30 Jan 2026cs.LGcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Model (LLM) inference is increasingly constrained by GPU memory capacity rather than compute throughput, driven by growing model sizes and the linear growth of the key-value (KV) cache during autoregressive decoding. Existing approaches mitigate memory pressure by offloading model state and KV tensors to host memory, but incur substantial latency due to limited PCIe bandwidth. We present Harvest, an opportunistic GPU cache management framework that exploits high-bandwidth peer-to-peer GPU interconnects to dynamically place model weights and KV cache in unused GPU memory. Harvest treats peer GPU memory as a transient cache tier, preserving correctness while reducing data movement overhead under dynamic memory availability. We demonstrate significant throughput speedup of more than 2 times by using Harvest to accelerate the retrieval of two widely-used inference components: expert layer weights and KV cache entries.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 28 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

20 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters13 Oct

Other references · 19

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  5. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  6. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  7. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  8. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  9. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cachecited by 48
  10. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  11. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  12. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domainscited by 14
7 more
  1. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputationcited by 12
  2. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanismcited by 9
  3. 2025HarMoEny: Efficient Multi-GPU Inference of MoE Modelscited by 7
  4. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placementcited by 7
  5. 2025Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensationcited by 3
  6. 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory Systemcited by 1
  7. 2026Joint Encoding of KV-Cache Blocks for Scalable LLM Servingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  2. 2026EMA: Elastic and Performance Transparent Memory Across GPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  2. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  3. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  4. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  5. 2024KVDirect: Distributed Disaggregated LLM Inference
  6. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
  7. 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems
  8. 2025Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching