mlsanityData to 14 Oct 2026

Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching

arXiv 2606.07684 ↗First posted 5 Jun 2026cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Disaggregated serving alleviates memory bottlenecks in Large Language Model (LLM) inference but creates a severe communication bottleneck: transmitting high-dimensional Key-Value (KV) caches often dominates time-to-first-token (TTFT). Moreover, reusing caches across heterogeneous models (e.g., base and fine-tuned variants) causes semantic misalignment that accumulates over layers, degrading generation quality. We propose Semantic Cache Distillation (SCD), a loss-constrained framework that replaces raw KV transmission with compact semantic codes. SCD addresses these challenges via two mechanisms: (1) Reuse, which reconstructs most layers from low-rank subspaces to minimize transfer cost, and (2) Patch, which predicts normalized inputs at sparse transition layers to truncate error propagation. Empirically, SCD delivers up to 2.65 × TTFT speedup over the oracle consumer prefill and dominates quantization and selective recomputation baselines on the quality--latency Pareto frontier in bandwidth-constrained regimes, while keeping generation quality within 5% F1 of the oracle.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

26 of 30 references.

References · 26

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  12. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
14 more
  1. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  2. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  3. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  5. 2020The Lipschitz Constant of Self-Attentioncited by 174
  6. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  7. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  8. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 61
  9. 2023How Smooth Is Attention?cited by 44
  10. 2024DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Servingcited by 35
  11. 2025TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMscited by 12
  12. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 11
  13. 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Schedulingcited by 10
  14. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026CacheBridge: Efficient Cross-Model KV Cache Transfercited by 6
  2. 2026Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reusecited by 3
  3. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0
  4. 2026LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replaycited by 0
  5. 2026To Cache or Not to Cache: Understanding and Mitigating Safety Incompatibility in Cross-Model KV Cache Reusecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  2. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  3. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  4. 2026SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
  5. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
  6. 2025Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity
  7. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  8. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference