mlsanityData to 7 Oct 2026

Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs

arXiv 2503.00979 ↗First posted 2 Mar 2025cs.CL cs.AI cs.LGcited by 20

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Autoregressive Transformers rely on Key-Value (KV) caching to accelerate inference. However, the linear growth of the KV cache with context length leads to excessive memory consumption and bandwidth constraints. This bottleneck is particularly problematic in real-time applications -- such as chatbots and interactive assistants -- where low latency and high memory efficiency are critical. Existing methods drop distant tokens or compress states in a lossy manner, sacrificing accuracy by discarding vital context or introducing bias. We propose MorphKV, an inference-time technique that maintains a constant-sized KV cache while preserving accuracy. MorphKV balances long-range dependencies and local coherence during text generation. It eliminates early-token bias while retaining high-fidelity context by adaptively ranking tokens through correlation-aware selection. Unlike heuristic retention or lossy compression, MorphKV iteratively refines the KV cache via lightweight updates guided by attention patterns of recent tokens. This approach captures inter-token correlation with greater accuracy, crucial for tasks like content creation and code generation. Our studies on long-response tasks show 52.9% memory savings and 18.2% higher accuracy on average compared to state-of-the-art prior works, enabling efficient real-world deployment.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 27 references that are in the map. First posted on arXiv 2 Mar 2025.

Builds on

24 of 27 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction4 Dec
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  3. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  5. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
  6. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  8. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget7 Apr
  9. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  10. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  11. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
3 more
  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 9

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  8. 2024LongGenBench: Long-context Generation Benchmarkcited by 27
  9. 2017Fast Amortized Inference and Learning in Log-linear Models with Randomly Perturbed Nearest Neighbor Searchcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 12.

20 papers.

Sort:

By area:

  1. 2025Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compressioncited by 72
  2. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  3. 2026MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokenscited by 20
  4. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  5. 2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoningcited by 17
  6. 2026WorldKV: Efficient World Memory with World Retrieval and Compressioncited by 14
  7. 2026Learning to Evict from Key-Value Cachecited by 12
  8. 2026LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generationcited by 12
  9. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  10. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  11. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
  12. 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Managementcited by 4
8 more
  1. 2025Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIMcited by 3
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inferencecited by 3
  3. 2026DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inferencecited by 3
  4. 2026FreqForcing: Autoregressive Long Video Generation via Spectral Self-Anchoringcited by 3
  5. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computingcited by 2
  6. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Servingcited by 2
  7. 2026Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgetscited by 1
  8. 2026GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fractional Decay KV-Cache: Ownership-Aware Memory Management for Improved Inference Relevancy in Dialog Systems
  2. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  3. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
  4. 2024RefreshKV: Updating Small KV Cache During Long-form Generation
  5. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  6. 2026CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
  7. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
  8. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference