mlsanityData to 14 Oct 2026

What It Costs to Compose, Rebuild, and Correct Precomputed Memory

arXiv 2608.30647 ↗First posted 31 Aug 2026cs.CL cs.LGcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under which it fails. Across experiments on Llama-3.1-8B-Instruct using both saved key-value caches and trained compressions of them, precomputed memory degrades when assembled from separately prepared parts, stays current only through rebuilds costing a large fraction of full preparation in our measurements, and ignores corrections served beside it conditional on phrasing. If precomputed memories can be served alongside one another, be cost-efficiently rebuilt, and be superseded by new information arriving in real-time, they can serve as a way to avoid re-feeding context to a model over repeated queries. The implication of our results for a deployed system that deals with a variety of queries is that precomputed memories are best rebuilt on the cadence at which new information changes what the memory was originally computed from. Both warm-rebuilding trained compressions of key-value caches and serving specifically-phrased updates beside a memory, as pasted text or injected cache state, show particular promise for keeping precomputed memories current, the latter as an interim measure between rebuilds, and we measure the cost and name the remaining questions associated with each.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 42 of its 52 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

42 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026InferScale: GPU-Native KV Injection for Personalized LLM Serving29 Jul
  2. 2026Cartridges at Scale: Training Modular KV Caches over Large Document Collections3 Jun
  3. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
26 more
  1. 2018Experience Replay for Continual Learningcited by 1,095
  2. 2022Mass-Editing Memory in a Transformercited by 1,001
  3. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  4. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  5. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  6. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  7. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  8. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  9. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
  10. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  11. 2022Parallel Context Windows for Large Language Modelscited by 81
  12. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 47
  13. 2021FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Searchcited by 38
  14. 2024Block-Attention for Efficient Prefillingcited by 37
  15. 2024LongHealth: A Question Answering Benchmark with Long Clinical Documentscited by 33
  16. 2025APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encodingcited by 29
  17. 2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26
  18. 2023Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deteriorationcited by 5
  19. 2026TEPA: Revoking Stale Memories for Conflict-Robust Language Agentscited by 5
  20. 2024Step Out and Seek Around: On Warm-Start Training with Incremental Datacited by 4
  21. 2026InjecMEM: Memory Injection Attack on LLM Agent Memory Systemscited by 4
  22. 2026ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agentscited by 3
  23. 2026When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memorycited by 3
  24. 2026Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Modelscited by 2
  25. 2026Utility Under Attack: Agent Memory Poisoning and the Limits of Content Screening and Provenance Rankingcited by 1
  26. 2026Towards Reliable, Generalizable, and Specific In-Context Knowledge Editing via Multi-Objective Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026PReM: Learning What to Preserve and When to Refresh for Context Compression
  2. 2026Positions Are Not Facts: The Mismatch Between KV Caches and Memory
  3. 2026Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability
  4. 2026Budgeted KV-Cache Repair for Evolving Contexts
  5. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolution
  6. 2025KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs
  7. 2026Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost
  8. 2026Correct Now, Insufficient Later: Auditing Update Sufficiency in Context Compression