mlsanityData to 7 Oct 2026

VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

arXiv 2605.17613 ↗First posted 17 May 2026cs.AR cs.LGcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The large size of the KV cache has become a major bottleneck for serving LLMs with increasing context lengths. In response, many KV cache compression methods, such as token dropping and quantization, have been proposed. However, almost all of these methods are inherently lossy-despite minimal accuracy degradation for short outputs, their outputs increasingly diverge from full-KV-cache outputs as more tokens are decoded, which leads to catastrophic failures in code generation and tool calling. We present VeriCache, the first inference framework that ensures the same output as full-KV-cache decoding but largely preserves the high decoding throughput of a range of KV cache compression algorithms. VeriCache uses the compressed KV cache to draft tokens, then verifies them against the full KV cache. While it may seem like just speculative decoding, VeriCache requires addressing a key system challenge to work-keeping the full KV cache out of GPU memory and minimizing the overhead of swapping it in for verification. The insight is two-fold: (1) compressed-KV decoding can be parallelized with full-KV swap, because one is HBM-bandwidth-bound and the other is PCIe/network-bound, and (2) the compressed KV cache often produces output similar to the full KV cache, allowing a long drafting horizon to amortize each full-KV swap. VeriCache applies to both long-context decoding and remote prefix caching, supports a broad family of token-dropping and quantization methods through a uniform compressor interface, and composes with traditional speculative decoding. Experimental results show that VeriCache achieves up to 4X higher throughput than full-KV inference while producing identical outputs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 75 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

47 of 75 references.

Closest ancestors · 17

Same direction, newest first

  1. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction25 Jan
  2. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruning12 Jan
  3. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  4. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  5. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching21 Sep
  6. 2025LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important7 Apr
  7. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention20 Feb
  8. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  9. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  10. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  12. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
5 more
  1. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  3. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  4. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  9. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  10. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  11. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  12. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
18 more
  1. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  2. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  3. 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 69
  4. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 67
  5. 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenariocited by 53
  6. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  7. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  8. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  9. 2024AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probabilitycited by 24
  10. 2025What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyondcited by 22
  11. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Modelscited by 13
  12. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  13. 2025DocR1: Evidence Page-Guided GRPO for Multi-Page Document Understandingcited by 11
  14. 2025PISanitizer: Preventing Prompt Injection to Long-Context LLMs via Prompt Sanitizationcited by 10
  15. 2024Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillationcited by 7
  16. 2025HydraRAG: Structured Cross-Source Enhanced Large Language Model Reasoningcited by 6
  17. 2025MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidancecited by 3
  18. 2025Multi-document Summarization through Multi-document Event Relation Graph Reasoning in LLMs: a case study in Framing Bias Mitigationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026What a Deletion Certificate Covers, and Where It Expires: Auditable Removal from a Support-Vector Memorycited by 2
  2. 2026A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forevercited by 2
  3. 2026Learning Agent Execution for KV-Cache Management in Agentic Servingcited by 1
  4. 2026Exact Record Omission in Delta Attention: A Transport Criterion, Its Cost, and a Replay Certificatecited by 0
  5. 2026Memory as a cache: Exact context reuse and deletion by constructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  2. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  3. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
  4. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
  5. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
  6. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  7. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  8. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference