# FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension

arXiv: https://arxiv.org/abs/2505.00570 · HTML: https://mlsanity.com/paper/2505.00570/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 1 May 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 30 of 30)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [Learning in the Frequency Domain](https://mlsanity.com/paper/2002.12416/) (2020)
- [InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory](https://mlsanity.com/paper/2402.04617/) (2024)
- [LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models](https://mlsanity.com/paper/2308.16137/) (2023)
- [Long Context Compression with Activation Beacon](https://mlsanity.com/paper/2401.03462/) (2024)
- [Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks](https://mlsanity.com/paper/2407.08454/) (2024)
- [ThinK: Thinner Key Cache by Query-Driven Pruning](https://mlsanity.com/paper/2407.21018/) (2024)
- [Theory of the Frequency Principle for General Deep Neural Networks](https://mlsanity.com/paper/1906.09235/) (2019)
- [Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction](https://mlsanity.com/paper/2409.17422/) (2024)
- [D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models](https://mlsanity.com/paper/2406.13035/) (2024)
- [LongGenBench: Long-context Generation Benchmark](https://mlsanity.com/paper/2410.04199/) (2024)
- [Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator](https://mlsanity.com/paper/2305.15099/) (2023)
- [LoCoCo: Dropping In Convolutions for Long Context Compression](https://mlsanity.com/paper/2406.05317/) (2024)
- [FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration](https://mlsanity.com/paper/2502.01068/) (2025)

## Built on by (most cited first, 7 of 7)

- [FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression](https://mlsanity.com/paper/2507.20030/) (2025)
- [Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models](https://mlsanity.com/paper/2508.06038/) (2025)
- [Adaptive KV Cache Reuse for Fast Long-Context LLM Serving](https://mlsanity.com/paper/2605.24022/) (2026)
- [Information-Aware KV Cache Compression for Long Reasoning](https://mlsanity.com/paper/2606.26875/) (2026)
- [UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inference](https://mlsanity.com/paper/2603.18446/) (2026)
- [Fre-Res: Frequency-Residual Video Token Compression for Efficient Video MLLMs](https://mlsanity.com/paper/2605.16366/) (2026)
- [Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches](https://mlsanity.com/paper/2607.20538/) (2026)

## Abstract (arXiv)

Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local information in both long prefilling and decoding scenarios. When extrapolating beyond the pretrained context length, their performance degrades sharply on long-context benchmarks. Motivated by the observation in the frequency domain that the context information is concentrated in the low-frequency components, we propose FreqKV, a parameter-free and architecture-agnostic approach. It iteratively compresses the increasing KV cache in the frequency domain, allowing models to process lengthy contexts efficiently. With minimal training at 8K length, FreqKV extends the context window of LLaMA-2-7B up to 256K tokens while maintaining stable perplexity. Extensive experiments across prefilling and decoding demonstrate that FreqKV enables robust context window extension and consistently outperforms existing KV cache compression methods on LLaMA-2 and LLaMA-3, highlighting its effectiveness for both understanding and generation in long contexts.