TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
Key-value (KV) caching is critical for efficient inference in large language models (LLMs), yet its memory footprint scales linearly with context length, resulting in a severe scalability bottleneck. Existing approaches largely treat KV states as equally important across time, implicitly assuming uniform precision and accessibility. However, this assumption contrasts with human memory systems, where memories vary in clarity, recall frequency, and relevance with temporal proximity.Motivated by this insight, we propose TTKV, a KV cache management framework that maps the human memory system onto the KV cache. TTKV partitions the KV cache into temporal tiers with heterogeneous capacity and precision. The design addresses three aspects: (1) Tier Layout, decoupling fast and slow memory using HBM and DRAM; (2) Tier Content, assigning more recent KV states to faster, higher-precision tiers based on temporal proximity; and (3) Tier Interaction, employing block-wise streaming attention to overlap communication and computation when accessing slow tiers. Experiments show that TTKV reduces cross-tier traffic by 5.94x on 128K-context tasks, achieving up to 76% latency reduction and 2x throughput improvement over strong baselines.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 18 references that are in the map. First posted on arXiv 27 Mar 2026.
Builds on
15 of 18 references.
Closest ancestors · 6
Same direction, newest first
- 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression30 Dec
- 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference19 May
- 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction4 Dec
- 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation26 Nov
- 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
- 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
Other references · 9
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2020Reformer: The Efficient Transformercited by 1,658
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
- 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
- 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
- 2025FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inferencecited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching
- 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
- 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
- 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
- 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
- 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
- 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
- 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization