mlsanityData to 14 Oct 2026

Low-Rank Key Value Attention

arXiv 2601.11471 ↗First posted 16 Jan 2026cs.LGcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The key-value (KV) cache is a primary memory bottleneck in Transformers. We propose Low-Rank Key-Value (LRKV) attention, which reduces KV cache memory by exploiting redundancy across attention heads, while being compute efficient. Each layer uses a shared full-rank KV projection augmented with low-rank, head-specific residuals, providing a continuous trade-off between complete sharing and full independence. After pretraining models of size 128M to 6.3B parameters, LRKV consistently achieves the lowest test loss among standard MHA, MQA/GQA, and MLA while using only 45-53% of MHA's KV cache. LRKV reaches equivalent baseline quality 18-25% faster (measured in training steps). After supervised midtraining, LRKV achieves the highest downstream task performance across ARC-Easy, ARC-Challenge, MMLU, GSM8K, and HumanEval benchmarks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 50 references that are in the map. First posted on arXiv 16 Jan 2026.

Builds on

39 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization26 May
  2. 2024Multi-matrix Factorization Attention26 Dec
  3. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  4. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
22 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  2. 2020Rethinking Attention with Performerscited by 1,601
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2018On the Spectral Bias of Neural Networkscited by 1,341
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  11. 2019Are Sixteen Heads Really Better than One?cited by 816
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  13. 2022CodeT: Code Generation with Generated Testscited by 384
  14. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  15. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
  16. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  17. 2025Tensor Product Attention Is All You Needcited by 31
  18. 2025KV-Distill: Nearly Lossless Learnable Context Compression for LLMscited by 19
  19. 2024Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learningcited by 14
  20. 2023Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Boundscited by 6
  21. 2024Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attentioncited by 6
  22. 2025QKV Projections Require a Fraction of Their Memorycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantizationcited by 0
  2. 2026Virtual Head Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
  2. 2026Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
  3. 2026AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD
  4. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding
  5. 2026LOOKAT: Lookup-Optimized Key-Attention for Memory-Efficient Transformers
  6. 2026Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration
  7. 2025EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
  8. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention