mlsanityData to 14 Oct 2026

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

arXiv 2410.21465 ↗First posted 28 Oct 2024cs.LG cs.CLcited by 134

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

With the widespread deployment of long-context large language models (LLMs), there has been a growing demand for efficient support of high-throughput inference. However, as the key-value (KV) cache expands with the sequence length, the increasing memory footprint and the need to access it for each token generation both result in low throughput when serving long-context LLMs. While various dynamic sparse attention methods have been proposed to speed up inference while maintaining generation quality, they either fail to sufficiently reduce GPU memory consumption or introduce significant decoding latency by offloading the KV cache to the CPU. We present ShadowKV, a high-throughput long-context LLM inference system that stores the low-rank key cache and offloads the value cache to reduce the memory footprint for larger batch sizes and longer sequences. To minimize decoding latency, ShadowKV employs an accurate KV selection strategy that reconstructs minimal sparse KV pairs on-the-fly. By evaluating ShadowKV on a broad range of benchmarks, including RULER, LongBench, and Needle In A Haystack, and models like Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K, and Qwen2-7B-128K, we demonstrate that it can support up to 6× larger batch sizes and boost throughput by up to 3.04× on an A100 GPU without sacrificing accuracy, even surpassing the performance achievable with infinite batch size under the assumption of infinite GPU memory. The code is available at https://github.com/bytedance/ShadowKV.

Key facts. Cited by 134 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 59 of its 70 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

59 of 70 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection16 Oct
  2. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  3. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  4. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  7. 2024Effectively Compress KV Heads for LLM11 Jun
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  10. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  11. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  12. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
4 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 43

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
31 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  4. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  11. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  12. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  13. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  14. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  15. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  16. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  17. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  18. 2024Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligencecited by 86
  19. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 73
  20. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  21. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  22. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  23. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 41
  24. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  25. 2023Parameter-efficient Tuning of Large-scale Multimodal Foundation Modelcited by 23
  26. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  27. 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 21
  28. 2024FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Modelcited by 19
  29. 2023TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewisecited by 15
  30. 2024Rhyme-aware Chinese lyric generator based on GPTcited by 10
  31. 2024LIVE: Learnable In-Context Vector for Visual Question Answeringcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 48; 2026: 83.

134 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  3. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 54
  4. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  5. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 40
  6. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
  7. 2025Tensor Product Attention Is All You Needcited by 31
  8. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  9. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  10. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 20
  11. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 19
  12. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference Systemcited by 17
38 more
  1. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 16
  2. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  3. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  4. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 13
  5. 2025TokenButler: Token Importance is Predictablecited by 12
  6. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 12
  7. 2025Streaming Attention Approximation via Discrepancy Theorycited by 11
  8. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 11
  9. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Cachingcited by 10
  10. 2025Hardware-Efficient Attention for Fast Decodingcited by 10
  11. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 10
  12. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelitycited by 10
  13. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9
  14. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMscited by 9
  15. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharingcited by 9
  16. 2026Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rolloutscited by 9
  17. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 9
  18. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMscited by 8
  19. 2026FASA: Frequency-aware Sparse Attentioncited by 8
  20. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  21. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  22. 2025HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inferencecited by 7
  23. 2025OjaKV: Context-Aware Online Low-Rank KV Cache Compressioncited by 7
  24. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6
  25. 2025SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruningcited by 6
  26. 2025SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matchingcited by 6
  27. 2025SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMscited by 6
  28. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoningcited by 6
  29. 2025Unveiling Simplicities of Attention: Adaptive Long-Context Head Identificationcited by 5
  30. 2025KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inferencecited by 5
  31. 2026KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planningcited by 5
  32. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  33. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inferencecited by 5
  34. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inferencecited by 5
  35. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 5
  36. 2025SAS: Simulated Attention Scorecited by 4
  37. 2025LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequencescited by 4
  38. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4

50 most cited of 134 All 134 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  2. 2026DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
  3. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
  4. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  5. 2024RefreshKV: Updating Small KV Cache During Long-form Generation
  6. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference
  7. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  8. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review