mlsanityData to 7 Oct 2026

RefreshKV: Updating Small KV Cache During Long-form Generation

arXiv 2411.05787 ↗First posted 8 Nov 2024cs.CLcited by 11

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Generating long sequences of tokens given a long-context input is a very compute-intensive inference scenario for large language models (LLMs). One prominent inference speed-up approach is to construct a smaller key-value (KV) cache, relieving LLMs from computing attention over a long sequence of tokens. While such methods work well to generate short sequences, their performance degrades rapidly for long-form generation. Most KV compression happens once, prematurely removing tokens that can be useful later in the generation. We propose a new inference method, RefreshKV, that flexibly alternates between full context attention and attention over a subset of input tokens during generation. After each full attention step, we update the smaller KV cache based on the attention pattern over the entire input. Applying our method to off-the-shelf LLMs achieves comparable speedup to eviction-based methods while improving performance for various long-form generation tasks. Lastly, we show that continued pretraining with our inference setting brings further gains in performance.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 32 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

26 of 32 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  2. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  7. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct

Other references · 18

  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  12. 20218-bit Optimizers via Block-wise Quantizationcited by 395
6 more
  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  3. 2021Efficient Attentions for Long Document Summarizationcited by 342
  4. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  5. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  6. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMscited by 8
  2. 2026LongFlow: Efficient KV Cache Compression for Reasoning Modelscited by 6
  3. 2025LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequencescited by 4
  4. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  5. 2025PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inferencecited by 1
  6. 2026SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answeringcited by 1
  7. 2026Random Attention: Rethinking KV Cache Eviction for Efficient Reasoningcited by 1
  8. 2026Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stabilitycited by 0
  9. 2026MentorPulse: Refreshing Cross-Model Latent Guidance for Long-Form Generationcited by 0
  10. 2026On-Demand Attention: Language Models Know When to Recallcited by 0
  11. 2026Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  2. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  3. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  7. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  8. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference