Models Take Notes at Prefill: KV Cache Can Be Editable and Composable
Not on the map. Why?
Abstract (arXiv)
Prefix caching reuses prefill only across an exactly shared prefix, so one changed field invalidates the entire downstream cache. Yet overwriting the field's own key/value vectors and reusing the rest leaves the model acting on the old value. The reason, established causally across four model families: at prefill the model has already written the field-conditioned conclusion onto downstream notes; the field's own key/value drives under 1% of the decision. Read as a notebook of memoized conclusions, two capabilities follow. (1) It is editable. A salient erratum amends the notes; and with chain-of-thought, editing the field alone recovers the decision (1.00 at 8B, ~1% compute), while without CoT it is ignored. (2) It is composable. The notes are position-portable, so a precompiled skill can be RoPE-repositioned and spliced into any context, indistinguishable from full recompute (logit cosine 0.90-0.999, twelve models) at O(L) rather than O(L^2) time-to-first-token. A unified edit+compose agent stays decision-identical to recompute at up to 14.9x lower latency. The approach applies to any per-token attention KV cache, validated across scale, quantization, Mixture-of-Experts, and multimodal caches, and extends to several attention variants through small adapters. Because the erratum is append-only, it composes with production prefix caching: in an online vLLM benchmark it keeps the prefix cache-aligned (98.5% hit-rate), cutting p90 time-to-first-token by 53-398x.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 14 Jun 2026.
Builds on
33 of 38 references.
References · 33
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2025Qwen3 Technical Reportcited by 8,820
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Mistral 7Bcited by 3,164
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
21 more
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
- 2022Editing Models with Task Arithmeticcited by 1,120
- 2022Mass-Editing Memory in a Transformercited by 1,001
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
- 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
- 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
- 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
- 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
- 2023Function Vectors in Large Language Modelscited by 308
- 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
- 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
- 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
- 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
- 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
- 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 47
- 2025MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Servingcited by 8
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KVcited by 2
- 2026Aborted but Not Forgotten: KV-Cache Retention Breaks Rollback Consistency in Language Agentscited by 0
- 2026Parametric Multimodal User Memory: Storing What Captions Cannot Carrycited by 0
- 2026Positions Are Not Facts: The Mismatch Between KV Caches and Memorycited by 0
- 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptationcited by 0
- 2026Budgeted KV-Cache Repair for Evolving Contextscited by 0
- 2026Budgeted KV-Cache Repair for Evolving Contexts12 Oct 2026
- 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptation29 Sep 2026
- 2026Positions Are Not Facts: The Mismatch Between KV Caches and Memory27 Sep 2026
- 2026Aborted but Not Forgotten: KV-Cache Retention Breaks Rollback Consistency in Language Agents16 Aug 2026
- 2026Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KV26 Jul 2026
- 2026Parametric Multimodal User Memory: Storing What Captions Cannot Carry8 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026Can I Buy Your KV Cache?
- 2026Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap
- 2026What It Costs to Compose, Rebuild, and Correct Precomputed Memory
- 2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
- 2026Written as a Record, Read as an Address: What a Forward Pass Leaves in an Operation's KV Cache
- 2025EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
- 2025KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs
- 2026PAGE: Partition-Aware Gated KV-Cache Eviction