Data to 7 Oct 2026

Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference

arXiv 2403.09054 ↗First posted 14 Mar 2024cs.LG cs.AI cs.ARcited by 124

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformers have emerged as the underpinning architecture for Large Language Models (LLMs). In generative language models, the inference process involves two primary phases: prompt processing and token generation. Token generation, which constitutes the majority of the computational workload, primarily entails vector-matrix multiplications and interactions with the Key-Value (KV) Cache. This phase is constrained by memory bandwidth due to the overhead of transferring weights and KV cache values from the memory system to the computing units. This memory bottleneck becomes particularly pronounced in applications that require long-context and extensive text generation, both of which are increasingly crucial for LLMs. This paper introduces "Keyformer", an innovative inference-time approach, to mitigate the challenges associated with KV cache size and memory bandwidth utilization. Keyformer leverages the observation that approximately 90% of the attention weight in generative inference focuses on a specific subset of tokens, referred to as "key" tokens. Keyformer retains only the key tokens in the KV cache by identifying these crucial tokens using a novel score function. This approach effectively reduces both the KV cache size and memory bandwidth usage without compromising model accuracy. We evaluate Keyformer's performance across three foundational models: GPT-J, Cerebras-GPT, and MPT, which employ various positional embedding algorithms. Our assessment encompasses a variety of tasks, with a particular emphasis on summarization and conversation tasks involving extended contexts. Keyformer's reduction of KV cache reduces inference latency by 2.1x and improves token generation throughput by 2.4x, while preserving the model's accuracy.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 66 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

46 of 66 references.

Closest ancestors · 4

Same direction, newest first

Other references · 42

2017Attention Is All You Needcited by 59,932
2020Language Models are Few-Shot Learnerscited by 32,199
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
2020Longformer: The Long-Document Transformercited by 3,362
2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
2023A Survey of Large Language Modelscited by 2,537
2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114

30 more

2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
2020Big Bird: Transformers for Longer Sequencescited by 1,673
2020Reformer: The Efficient Transformercited by 1,658
2020Rethinking Attention with Performerscited by 1,584
2019Generating Long Sequences with Sparse Transformerscited by 1,523
2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
2022Efficiently Scaling Transformer Inferencecited by 529
2023Learning to Compress Prompts with Gist Tokenscited by 379
2021Efficient Attentions for Long Document Summarizationcited by 342
2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
2019Adaptive Attention Span in Transformerscited by 148
2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attentioncited by 44
2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 38
2021Accelerating Recommendation System Training by Leveraging Popular Choicescited by 22
2023FLuID: Mitigating Stragglers in Federated Learning using Invariant Dropoutcited by 16
2022Heterogeneous Acceleration Pipeline for Recommendation System Trainingcited by 13
2021EL-Attention: Memory Efficient Lossless Attention for Generationcited by 6
2023Ad-Rec: Advanced Feature Interactions to Address Covariate-Shifts in Recommendation Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 53; 2026: 38.

124 papers.

Sort:

By area:

2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 71
2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 58
2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46

38 more

2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 41
2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
2025Cognitive Memory in Large Language Modelscited by 28
2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
2025Tensor Product Attention Is All You Needcited by 27
2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMscited by 20
2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoningcited by 17
2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
2025MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generationcited by 14
2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 13
2024XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inferencecited by 12
2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
2026Learning to Evict from Key-Value Cachecited by 12
2024Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Modelscited by 11
2024RefreshKV: Updating Small KV Cache During Long-form Generationcited by 11
2024Cross-Self KV Cache Pruning for Efficient Vision-Language Inferencecited by 11
2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Managementcited by 11
2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
2025Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inferencecited by 10
2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Modelscited by 9
2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inferencecited by 8
2024A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decodercited by 7
2024AdaCM2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reductioncited by 7
2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Managementcited by 7
2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inferencecited by 7
2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way