mlsanityData to 14 Oct 2026

MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection

arXiv 2410.14731 ↗First posted 16 Oct 2024cs.LG cs.AI cs.CLcited by 29

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV cache has become a de facto technique for the inference of large language models (LLMs), where tensors of shape (layer number, head number, sequence length, feature dimension) are introduced to cache historical information for self-attention. As the size of the model and data grows, the KV cache can quickly become a bottleneck within the system in both storage and memory transfer. To address this, prior studies usually focus on the first three axes of the cache tensors for compression. This paper supplements them, focusing on the feature dimension axis, by utilizing low-rank projection matrices to transform the cache features into spaces with reduced dimensions. We begin by investigating the canonical orthogonal projection method for data compression through principal component analysis (PCA). We observe the issue with PCA projection where significant performance degradation is observed at low compression rates. To bridge the gap, we propose to directly tune the orthogonal projection matrices with a distillation objective using an elaborate Matryoshka training strategy. After training, we adaptively search for the optimal compression rates for various layers and heads given varying compression budgets. Compared to previous works, our method can easily embrace pre-trained LLMs and hold a smooth tradeoff between performance and compression rate. We empirically witness the high data efficiency of our training procedure and find that our method can sustain over 90% performance with an average KV cache compression rate of 60% (and up to 75% in certain extreme scenarios) for popular LLMs like LLaMA2-7B-base and Mistral-7B-v0.3-base.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 42 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

32 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  2. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  3. 2024GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression16 Jul
  4. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  6. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
14 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  4. 2023A Comprehensive Overview of Large Language Modelscited by 648
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  6. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  7. 2022Matryoshka Representation Learningcited by 282
  8. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  9. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  10. 2023Continual Pre-training of Language Modelscited by 198
  11. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  12. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  13. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  14. 2024From LLM to NMT: Advancing Low-Resource Machine Translation with Claudecited by 58

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 19.

29 papers.

Sort:

By area:

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  3. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 36
  4. 2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibrationcited by 16
  5. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11
  6. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 9
  7. 2025NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cachecited by 7
  8. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  9. 2025OjaKV: Context-Aware Online Low-Rank KV Cache Compressioncited by 7
  10. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  11. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
  12. 2026Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifoldcited by 3
17 more
  1. 2026RAP: KV-Cache Compression via RoPE-Aligned Pruningcited by 3
  2. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 2
  3. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  4. 2026Hyperspectral Image Fusion with Spectral-Band and Fusion-Scale Agnosticismcited by 1
  5. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 1
  6. 2026Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scalescited by 0
  7. 2026Rotation-Aligned Key Channel Pruning for Efficient Vision-Language Model Inferencecited by 0
  8. 2026Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understandingcited by 0
  9. 2026Dual Dimensionality for Local and Global Attentioncited by 0
  10. 2026VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMscited by 0
  11. 2026Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Cachescited by 0
  12. 2026DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocationcited by 0
  13. 2026VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branchcited by 0
  14. 2026MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compressioncited by 0
  15. 2026DimPO: Dimensionality Reduction for Attention using Preference Optimizationcited by 0
  16. 2026Telescopic Language Modelscited by 0
  17. 2026AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVDcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  2. 2026One Size Does Not Fit All: Token-Wise Adaptive Compression for KV Cache
  3. 2025KV-CAR: KV Cache Compression using Autoencoders and KV Reuse in Large Language Models
  4. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
  5. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty
  6. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy
  7. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs