NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
Long-context language models are limited by the memory footprint of the key-value (KV) cache. Existing training-free KV compression methods usually rank tokens by one importance signal -- attention, recency, layer-wise allocation, or key distinctiveness -- which becomes brittle when useful context is globally distinctive, locally episodic, or immediately relevant. We introduce NestedKV, a key-only KV cache compression method inspired by the Continuum Memory System in Nested Learning. NestedKV maintains global, block-level, and sliding-window key anchors, scores tokens by multi-time-scale cosine anomaly, and combines the resulting rankings with a training-free outer learner using head-adaptive mixing and surprise-gated token routing. The score is paired with adaptive per-head budgets and requires no training or LLM modification. Across RULER (4k--32k), LooGLE, LongBench, LongBench-E, InfiniteBench, and MMLU-Pro on Qwen3 and Llama-3.2 models, NestedKV is strongest when the retained cache is small. On Qwen3-4B, it improves over KeyDiff by up to 19.10 points on RULER and 19.29 on LongBench at r=0.75; at r=0.95, it retains 37.32 on LongBench versus 17.55 for KeyDiff.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 21 references that are in the map. First posted on arXiv 26 May 2026.
Builds on
16 of 21 references.
Closest ancestors · 6
Same direction, newest first
- 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
- 2025FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management21 May
- 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
- 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
- 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
- 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May
Other references · 10
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
- 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
- 2024Titans: Learning to Memorize at Test Timecited by 303
- 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
- 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
- 2026SONIC: Segmented Optimized Nexus for Information Compression in Key-Value Cachingcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
- 2026GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
- 2026Behavior-Preserving KV Cache Compression
- 2025FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension
- 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
- 2026Training Transformers for KV Cache Compressibility
- 2026More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression
- 2024CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios