mlsanityData to 7 Oct 2026

KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding

arXiv 2507.11273 ↗First posted 15 Jul 2025cs.CLcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) based on Transformer Decoders have become the preferred choice for conversational generative AI. Despite the overall superiority of the Decoder architecture, the gradually increasing Key-Value (KV) cache during inference has emerged as a primary efficiency bottleneck, both in aspects of memory consumption and data transfer bandwidth limitations. To address these challenges, we propose a paradigm called KV-Latent. By down-sampling the Key-Value vector dimensions into a latent space, we can significantly reduce the KV Cache footprint and improve inference speed, only with a small amount of extra training, less than 1% of pre-training takes. Besides, we enhanced the stability of Rotary Positional Embedding applied on lower-dimensional vectors by modifying its frequency sampling mechanism, avoiding noise introduced by higher frequencies while retaining position attenuation. Our experiments, including both models with Grouped Query Attention and those without, have yielded satisfactory results. Finally, we conducted comparative experiments to study the impact of separately reducing Key and Value components on model's performance. Our approach allows for the construction of more efficient language model systems, and opens the new possibility on KV Cache saving and efficient LLMs. Our code is available at https://github.com/ShiLuohe/KV-Latent.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 39 references that are in the map. First posted on arXiv 15 Jul 2025.

Builds on

26 of 39 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers9 Jul
  2. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy4 Oct
  3. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  4. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  5. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  6. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
6 more
  1. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  2. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  3. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  4. 2024SirLLM: Streaming Infinite Retentive LLMcited by 16
  5. 2023Centroid-centered Modeling for Efficient Vision Transformer Pre-trainingcited by 1
  6. 2025Label Drop for Multi-Aspect Relation Modeling in Universal Information Extractioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Modelscited by 4
  2. 2025XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compressioncited by 4
  3. 2026WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantizationcited by 3
  4. 2026From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizonscited by 1
  5. 2026Efficient Matrix Implementation for Rotary Position Embeddingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SALS: Sparse Attention in Latent Space for KV cache Compression
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
  3. 2024Effectively Compress KV Heads for LLM
  4. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
  5. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  6. 2025KV-CAR: KV Cache Compression using Autoencoders and KV Reuse in Large Language Models
  7. 2025TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering
  8. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM