# Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache

arXiv: https://arxiv.org/abs/2506.11886 · HTML: https://mlsanity.com/paper/2506.11886/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- First posted on arXiv 13 Jun 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Built on by (most cited first, 5 of 5)

- [LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs](https://mlsanity.com/paper/2506.14429/) (2025)
- [CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing](https://mlsanity.com/paper/2508.16134/) (2025)
- [Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs](https://mlsanity.com/paper/2512.07525/) (2025)
- [VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization](https://mlsanity.com/paper/2603.16435/) (2026)
- [When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon](https://mlsanity.com/paper/2605.05699/) (2026)

## Abstract (arXiv)

Large Language Models struggle with memory demands from the growing Key-Value (KV) cache as context lengths increase. Existing compression methods homogenize head dimensions or rely on attention-guided token pruning, often sacrificing accuracy or introducing computational overhead. We propose FourierAttention, a training-free framework that exploits the heterogeneous roles of transformer head dimensions: lower dimensions prioritize local context, while upper ones capture long-range dependencies. By projecting the long-context-insensitive dimensions onto orthogonal Fourier bases, FourierAttention approximates their temporal evolution with fixed-length spectral coefficients. Evaluations on LLaMA models show that FourierAttention achieves the best long-context accuracy on LongBench and Needle-In-A-Haystack (NIAH). Besides, a custom Triton kernel, FlashFourierAttention, is designed to optimize memory via streamlined read-write operations, enabling efficient deployment without performance compromise.