mlsanityData to 14 Oct 2026

SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference

arXiv 2508.02751 ↗First posted 3 Aug 2025cs.LG cs.AIcited by 6

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV cache eviction has emerged as an effective solution to alleviate resource constraints faced by LLMs in long-context scenarios. However, existing token-level eviction methods often overlook two critical aspects: (1) their irreversible eviction strategy fails to adapt to dynamic attention patterns during decoding (the saliency shift problem), and (2) they treat both marginally important tokens and truly unimportant tokens equally, despite the collective significance of marginal tokens to model performance (the marginal information over-compression problem). To address these issues, we design two compensation mechanisms based on the high similarity of attention matrices between LLMs of different scales. We propose SmallKV, a small model assisted compensation method for KV cache compression. SmallKV can maintain attention matching between different-scale LLMs to: 1) assist the larger model in perceiving globally important information of attention; and 2) use the smaller model's attention scores to approximate those of marginal tokens in the larger model. Extensive experiments on benchmarks including GSM8K, BBH, MT-Bench, and LongBench demonstrate the effectiveness of SmallKV. Moreover, efficiency evaluations show that SmallKV achieves 1.75 - 2.56 times higher throughput than baseline methods, highlighting its potential for efficient and performant LLM inference in resource constrained environments.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 55 references that are in the map. First posted on arXiv 3 Aug 2025.

Builds on

46 of 55 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty12 Dec
  2. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  3. 2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference30 Oct
  4. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing24 Oct
  5. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate15 Oct
  6. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  7. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  8. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  9. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  10. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  11. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
7 more
  1. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  2. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  3. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  4. 2024LoMA: Lossless Compressed Memory Attention16 Jan
  5. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
15 more
  1. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  2. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  3. 2023Speculative Decoding with Big Little Decodercited by 164
  4. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  5. 2024Transformers are Multi-State RNNscited by 120
  6. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  7. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  8. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  9. 2024What Matters in Transformers? Not All Attention is Neededcited by 84
  10. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  11. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  12. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  13. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57
  14. 2023Compressed Context Memory For Online Language Model Interactioncited by 43
  15. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarioscited by 3
  2. 2026TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Taskscited by 3
  3. 2026TrigReason: Trigger-Based Collaboration between Small and Large Reasoning Modelscited by 1
  4. 2026ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inferencecited by 1
  5. 2026Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decodingcited by 0
  6. 2026Emergent Structure in the Marginal Attention Space of Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  2. 2026From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
  3. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  4. 2024MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache
  5. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  6. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  7. 2026KVSculpt: KV Cache Compression as Distillation
  8. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget