mlsanityData to 7 Oct 2026

FDC: Fast KV Dimensionality Compression for Efficient LLM Inference

arXiv 2408.04107 ↗First posted 7 Aug 2024cs.LG cs.DCcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

In large-language models, memory constraints in the Key-Value Cache (KVC) pose a challenge during inference. In this work, we propose FDC, a fast KV dimensionality compression system that eliminates the decompression overhead incurred in the existing KV dimensionality compression system, Palu, and reduces attention time. Moreover, FDC employs adaptive compression, tailoring KV compression rates across heads and layers based on their contributions to inference to maximize overall compression while maintaining an accuracy loss constraint. Additionally, FDC enhances the attention kernel to balance the uneven workloads caused by the adaptive compression approach to further reduce attention computation latency. Comprehensive experiments demonstrate that compared to Palu, FDC can reduce Job Completion Time (JCT) by up to 64%, and delivers up to 1.97X throughput under the same latency, while maintaining 99% of the accuracy without compression. When state-of-the-art eviction and quantization methods are combined with FDC, they exhibit similar improvements compared to those combined with Palu. We open-sourced the code.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 44 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

27 of 44 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  2. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  3. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  4. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  5. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  6. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  10. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  12. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
5 more
  1. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  2. 2019DocBERT: BERT for Document Classificationcited by 68
  3. 2024Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integrationcited by 17
  4. 2022Source Code Summarization with Structural Relative Position Guided Transformercited by 14
  5. 2024Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Agent-X: Full Pipeline Acceleration of On-device AI Agentscited by 4
  2. 2026Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifoldcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026KVSculpt: KV Cache Compression as Distillation
  2. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  3. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  4. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  5. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  6. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  7. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
  8. 2024Effectively Compress KV Heads for LLM