mlsanityData to 7 Oct 2026

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression

arXiv 2408.05646 ↗First posted 10 Aug 2024cs.LG cs.AI cs.CLcited by 71

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Large language models (LLMs) represent a groundbreaking advancement in the domain of natural language processing due to their impressive reasoning abilities. Recently, there has been considerable interest in increasing the context lengths for these models to enhance their applicability to complex tasks. However, at long context lengths and large batch sizes, the key-value (KV) cache, which stores the attention keys and values, emerges as the new bottleneck in memory usage during inference. To address this, we propose Eigen Attention, which performs the attention operation in a low-rank space, thereby reducing the KV cache memory overhead. Our proposed approach is orthogonal to existing KV cache compression techniques and can be used synergistically with them. Through extensive experiments over OPT, MPT, and Llama model families, we demonstrate that Eigen Attention results in up to 40% reduction in KV cache sizes and up to 60% reduction in attention operation latency with minimal drop in performance. Code is available at https://github.com/UtkarshSaxena1/EigenAttn.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 44 references that are in the map. First posted on arXiv 10 Aug 2024.

Builds on

27 of 44 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
15 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  2. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  4. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  5. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  6. 2021Gradient Projection Memory for Continual Learningcited by 338
  7. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  8. 2022Language model compression with weighted low-rank factorizationcited by 192
  9. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  10. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  11. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  12. 2024Long Context Compression with Activation Beaconcited by 101
  13. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
  14. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  15. 2024ATP: Enabling Fast LLM Serving via Attention on Top Principal Keyscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 26; 2026: 41.

71 papers.

Sort:

By area:

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  3. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  4. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
  5. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  6. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
  7. 2025Inference-Time Hyper-Scaling with KV Cache Compressioncited by 28
  8. 2025Tensor Product Attention Is All You Needcited by 27
  9. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMscited by 20
  10. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  11. 2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibrationcited by 16
  12. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 13
38 more
  1. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  2. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11
  3. 2025Hardware-Efficient Attention for Fast Decodingcited by 10
  4. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 9
  5. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelitycited by 9
  6. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  7. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 8
  8. 2026Value-Aware Stochastic KV Cache Eviction for Reasoning Modelscited by 8
  9. 2025SALS: Sparse Attention in Latent Space for KV cache Compressioncited by 7
  10. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6
  11. 2025FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Managementcited by 6
  12. 2025LatentLLM: Attention-Aware Joint Tensor Compressioncited by 6
  13. 2025OjaKV: Context-Aware Online Low-Rank KV Cache Compressioncited by 6
  14. 2026FASA: Frequency-aware Sparse Attentioncited by 6
  15. 2025KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embeddingcited by 5
  16. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerializationcited by 4
  17. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  18. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inferencecited by 4
  19. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Servingcited by 4
  20. 2025μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Expertscited by 3
  21. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  22. 2026Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifoldcited by 3
  23. 2026Learning When to Attend: Conditional Memory Access for Long-Context LLMscited by 3
  24. 2026FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fastcited by 2
  25. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  26. 2026Low-Rank Key Value Attentioncited by 1
  27. 2026TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Flycited by 1
  28. 2026OASIS: Online Activation Subspace Learning for Memory-Efficient Trainingcited by 1
  29. 2026DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisionscited by 1
  30. 2026LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decodingcited by 1
  31. 2026SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Codingcited by 1
  32. 2026KV Cache Compression Through the Lens of Transform Codingcited by 1
  33. 2026CompKV: Compensation-Aware KV Selection for Long-Context LLM Inferencecited by 1
  34. 2025ASTRA: Communication-Efficient Acceleration for Multi-Device Transformer Inferencecited by 0
  35. 2025AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMscited by 0
  36. 2025SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compressioncited by 0
  37. 2026Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compressioncited by 0
  38. 2026ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrievalcited by 0

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
  2. 2024Effectively Compress KV Heads for LLM
  3. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
  4. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  5. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  6. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
  8. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head