mlsanityData to 7 Oct 2026

FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration

arXiv 2502.01068 ↗First posted 3 Feb 2025cs.LG cs.CLcited by 13

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

While large language models (LLMs) excel at handling long-context sequences, they require substantial prefill computation and key-value (KV) cache, which can heavily burden computational efficiency and memory usage in both prefill and decoding stages. Recent works that compress KV caches with prefill acceleration reduce this cost but inadvertently tie the prefill compute reduction to the decoding KV budget. This coupling arises from overlooking the layer-dependent variation of critical context, often leading to accuracy degradation. To address this issue, we introduce FastKV, a KV cache compression framework designed to reduce latency in both prefill and decoding by leveraging the stabilization of token importance in later layers. FastKV performs full-context computation until a Token-Selective Propagation (TSP) layer, which forwards only the most informative tokens to subsequent layers. From these propagated tokens, FastKV independently selects salient KV entries for caching, thereby decoupling KV budget from the prefill compute reduction based on the TSP decision. This independent control of the TSP rate and KV retention rate enables flexible optimization of efficiency and accuracy. Experimental results show that FastKV achieves speedups of up to 1.82× in prefill and 2.87× in decoding compared to the full-context baseline, while matching the accuracy of the decoding-only baselines. Our code is available at https://github.com/dongwonjo/FastKV.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 34 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

23 of 34 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  3. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction25 Sep
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  6. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  8. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 14

  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  12. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
2 more
  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025FreqKV: Key-Value Compression in Frequency Domain for Context Window Extensioncited by 7
  2. 2025CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inferencecited by 7
  3. 2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challengescited by 3
  4. 2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inferencecited by 3
  5. 2026Information-Aware KV Cache Compression for Long Reasoningcited by 2
  6. 2026Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selectioncited by 1
  7. 2026UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inferencecited by 0
  8. 2026SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Modelscited by 0
  9. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0
  10. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 0
  11. 2026SALT: Salience-Aware Lexical Trie for Long-Context Compressioncited by 0
  12. 2026Parser States Already Know: Structure-Conditioned KV Persistence for Structured Generationcited by 0
1 more
  1. 2026CONDUIT: A Unified Residual-Stream Restoration Framework for KV Cache Reuse in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  2. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
  3. 2025LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important
  4. 2026DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
  5. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  6. 2024CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios
  7. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  8. 2026HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding