mlsanityData to 14 Oct 2026

KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs

arXiv 2512.11851 ↗First posted 4 Dec 2025cs.LG cs.AI cs.NEcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Whether attention key value (KV) states computed for one prompt for a small LLM can be reused to accelerate inference on a new similar prompt, giving an increase to the space to its context memory using an approach called token recycling. Using a standard Hugging Face setup with DialoGPT-medium (a 345M parameter GPT-2 style decoder trained on 147M Reddit exchanges, 2005 to 2017) as the testbed, we build a cache of past activations and get entries by sentence embeddings, then reuse cached past key values when the cached prompt is an exact prefix of the new input. We compare recycled vs. baseline runs on latency and output fidelity, and log reuse depth in tokens. Reproducibility requires no model modifications, cached KVs are serialized to the CPU, reloaded, and supplied to the generate function to continue decoding from the cached prefix. In tests, we observe consistent speedups when prefix overlap exists, with no material degradation in output semantics, and when overlap is absent, behavior matches baseline.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 7 of its 15 references that are in the map. First posted on arXiv 4 Dec 2025.

Builds on

7 of 15 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  2. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference11 Mar
  3. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression19 Feb
  4. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  5. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 1

  1. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Probing the Prompt KV Cache: Where It Becomes Dispensable
  2. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
  3. 2024RefreshKV: Updating Small KV Cache During Long-form Generation
  4. 2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
  5. 2025Retrospective Sparse Attention for Efficient Long-Context Generation
  6. 2026KV Cache Offloading for Context-Intensive Tasks
  7. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
  8. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse